Durable Heavy Infra

Overview

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” — Antoine de Saint-Exupéry

This insight from Antoine de Saint-Exupéry reflects the philosophy behind Durable Heavy Infra. We observe that the landscape of infrastructure management has evolved significantly, moving from manual operations to comprehensive, often monolithic, infrastructure-as-code frameworks.

Durable Heavy Infra, however, introduces a collection of focused primitives for this domain. Unlike traditional monolithic frameworks that abstract away underlying details, our approach prioritizes granular control and composability.

Each tool is purpose-built for a specific service and supports multiple operational modes, facilitating integration into diverse workflows. This design emphasizes reliability and composability, acknowledging that different contexts demand varying levels of abstraction and control.

Key Characteristics

• Service-Specific: Each tool is designed to manage one particular infrastructure service. This focused approach typically enhances effectiveness by concentrating on a single domain. This often contributes to improved understanding, maintainability, and security within that specific service context.
• Multi-Modal Operation: Infrastructure management occurs in diverse environments. Our tools, therefore, support various operational modes. These include a Command Line Interface (CLI), JSON Lines (JSONL) pipes for scripting, a REST API for service integration, and NATS messaging for event-driven architectures. This high-performance distributed messaging system further enhances flexibility, facilitating seamless integration into different operational contexts.
• Declarative and Imperative: Our tools support both querying the current state of your infrastructure (declarative) and directly modifying it (imperative). This dual capability empowers users to inspect and understand their environment, and to enact precise changes when needed. You might ask: how does this approach differ from purely declarative, monolithic infrastructure-as-code solutions? While we support both paradigms, our focus on composable primitives offers a distinct approach, providing more granular control, though at the cost of higher-level abstraction.
• Composable: Adhering to a Unix-style philosophy, each tool is designed to perform a single task effectively. These tools can, therefore, be chained to construct more complex infrastructure workflows. This approach to composability provides flexibility, allowing tools to be combined in novel ways. It also reduces vendor lock-in by enabling the swapping of components or integration with diverse tools within an ecosystem.

Design Principles

Reliability First

We prioritize reliability to ensure your infrastructure remains stable and predictable. We achieve this through:

• Conservative approach to state changes: We minimize unintended side effects by requiring explicit actions for modifications and providing clear previews of changes.
• Comprehensive error handling and rollback capabilities: Our tools are designed to gracefully handle failures, offering mechanisms to revert to a known good state when issues arise.
• Thorough validation before executing modifications: Before any changes are applied, inputs are rigorously validated to prevent misconfigurations and ensure operational integrity.

Clear Communication

Understanding your infrastructure’s state and the impact of your actions is crucial. Our tools facilitate this through:

• Explicit command structure: Commands are designed to be unambiguous, making their intent and effects clear.
• Detailed logging and status reporting: Comprehensive logs and status updates provide visibility into operations, aiding in auditing and debugging.
• Human-readable output formats: Information is presented in formats that are easy for humans to parse and understand, reducing cognitive load.

Composability

Following the Unix philosophy, each tool is designed to perform a single task effectively, enabling you to build complex workflows from simpler components.

• Unix-style tool philosophy: do one thing well: Each primitive focuses on a specific function, ensuring clarity and reducing complexity.
• Standard input/output formats for easy chaining: Tools communicate using consistent data formats, allowing them to be easily piped and combined.
• Consistent interface patterns across all tools: A uniform interface reduces the learning curve and promotes seamless integration across the Durable Heavy Infra ecosystem.

Long-term Sustainability

We design for the future, ensuring that your infrastructure management solutions remain viable and adaptable over time.

• Stable APIs that won’t break existing workflows: We commit to API stability, minimizing the need for costly refactoring as tools evolve.
• Clear documentation and examples: Comprehensive and up-to-date documentation, along with practical examples, ensures ease of use and understanding.
• Modular design for easy maintenance and updates: The architecture allows for independent updates and extensions, simplifying maintenance and facilitating future enhancements.

Installation

To get started with Durable Heavy Infra, you’ll need to install the tools. The specific installation method may vary depending on your operating system and preferred package manager.

Prerequisites

• [Prerequisite 1]: [Description]
• [Prerequisite 2]: [Description]

Installation Steps

Using a Package Manager (Recommended)

For most users, installing via a package manager is the simplest approach.

# Placeholder: Replace with actual installation command, e.g.,
# $ brew install durable-heavy-infra
# $ apt install durable-heavy-infra
$ [your-package-manager] install durable-heavy-infra

Building from Source

If you prefer to build from source, follow these steps:

1. Clone the repository:

   $ git clone https://github.com/durableprogramming/durable-heavy-infra.git
   $ cd durable-heavy-infra
   

2. Build the tools:

   # Placeholder: Replace with actual build command, e.g.,
   # $ cargo build --release
   # $ make install
   $ [your-build-command]
   

Architecture

To achieve reliable and composable infrastructure management, Durable Heavy Infra employs a specific architectural design. This section details the fundamental structure of its tools and the various execution modes they support.

Tool Structure

Each infrastructure tool provides:

• Query Operations: Retrieve current state (“list all instances”, “show instances with tag X”)
• Modification Operations: Change infrastructure state (“stop instance Y”, “create N instances of type Z”)

Execution Modes

Durable Heavy Infra tools are designed for flexibility, supporting multiple execution modes to integrate seamlessly into various operational contexts. In all examples below, tool-name is a placeholder for the specific infrastructure tool you are using (e.g., instance-tool, storage-tool).

CLI Mode For direct human interaction and shell scripting, the Command Line Interface (CLI) provides a familiar and powerful way to manage your infrastructure. This mode is ideal for interactive tasks and simple automation.

$ instance-tool list --tag production

This command uses instance-tool to list all EC2 instances tagged with “production”.

$ instance-tool stop i-0abcdef1234567890

This command uses instance-tool to stop a specific EC2 instance by its ID.

[
  { "id": "i-0abcdef1234567890", "state": "running", "tags": { "name": "web-server-1", "env": "production" } },
  { "id": "i-0fedcba9876543210", "state": "running", "tags": { "name": "app-server-1", "env": "production" } }
]

JSONL Pipe Mode For automated scripting and chaining commands in a Unix-like fashion, JSON Lines (JSONL), a format where each line is a valid JSON object, allows you to pass structured data between tools.

$ echo '{"command": "list", "filters": {"tag": "production"}}' | instance-tool --jsonl

This command pipes a JSONL object, specifying the “list” command and “production” tag filter, to instance-tool to retrieve production instances.

{"id": "i-0abcdef1234567890", "state": "running", "tags": { "name": "web-server-1", "env": "production" }}
{"id": "i-0fedcba9876543210", "state": "running", "tags": { "name": "app-server-1", "env": "production" }}

REST API Mode When integrating with web applications, dashboards, or other services, the REST API mode allows tools to expose a standard HTTP interface. Choose this mode for programmatic access from external systems and microservices.

$ instance-tool --api --port 8080 &

This command starts the instance-tool in API mode, listening on port 8080, and runs it in the background.

INFO: instance-tool API server started on port 8080

The exact output may vary slightly (e.g., including timestamps or process IDs), but it confirms the API server is running and ready to accept requests.

$ curl -X POST http://localhost:8080/command \
  -H "Content-Type: application/json" \
  -d '{\"command\": \"list\", \"filters\": {\"tag\": \"production\"}}'

This curl command sends a POST request to the instance-tool API, which is running locally on port 8080, to execute the “list” command with a filter for “production” tagged instances.

[
  { "id": "i-0abcdef1234567890", "state": "running", "tags": { "name": "web-server-1", "env": "production" } },
  { "id": "i-0fedcba9876543210", "state": "running", "tags": { "name": "app-server-1", "env": "production" } }
]

The API response mirrors the structured JSON output of the CLI mode, ensuring a consistent data representation across different interaction methods. This consistency simplifies integration with various clients and services.

NATS Mode For event-driven architectures and distributed systems, NATS (Neural Asyncronous Transfer System), a high-performance publish-subscribe messaging system, provides a lightweight and efficient communication bus for sending commands and receiving responses. NATS is often used for microservices communication, IoT messaging, and general-purpose distributed messaging. This mode facilitates asynchronous communication and real-time responsiveness, making it ideal for scenarios where immediate feedback or tightly coupled synchronous operations are not required, and a publish-subscribe model is beneficial.

$ instance-tool --nats --subject infrastructure.commands &

This command starts the instance-tool in NATS mode, enabling it to subscribe to the infrastructure.commands subject for event-driven operations, and runs it in the background.

INFO: instance-tool NATS subscriber started on subject 'infrastructure.commands'

Common Considerations

When working with infrastructure management tools, it’s important to consider potential challenges and best practices. You might wonder about the trade-offs of our granular and composable approach. While it offers significant flexibility and control, it inherently introduces complexities in orchestration and overall system visibility compared to more opinionated, monolithic solutions. To address these trade-offs and ensure robust infrastructure management, we emphasize the following best practices. Many of these are directly supported by Durable Heavy Infra’s design principles:

• Idempotency: Ensure that repeated execution of modification operations yields the same result without unintended side effects. Our tools are designed with idempotency in mind, but complex workflows built by chaining tools should also account for this.
• Permissions: Grant only the necessary permissions (least privilege) to the tools and the identities executing them. Misconfigured permissions can lead to unintended access or modifications.
• State Management: Understand how state is managed, especially in distributed or concurrent environments. While Durable Heavy Infra focuses on direct interaction, external state management (e.g., version control for configurations) is crucial for complex deployments.
• Testing: Thoroughly test your infrastructure automation scripts and workflows in non-production environments before deploying to production.

Configuration and Versioning

Configuration Options

Durable Heavy Infra tools can be configured through various means, including command-line flags, environment variables, and configuration files. Specific configuration options are detailed in the documentation for each individual tool.

• Command-Line Flags: Most tools support flags for immediate control (e.g., --port, --jsonl).
• Environment Variables: For persistent settings, environment variables can be used (e.g., DURABLE_INFRA_API_KEY).
• Configuration Files: (Placeholder: If applicable, mention default config file paths and formats, e.g., ~/.config/durable-infra/config.yaml)

Further Exploration and Resources

We encourage you to experiment with Durable Heavy Infra’s tools to discover their full potential. While this overview provides a foundational understanding, the specifics of each service-specific tool (e.g., instance-tool, storage-tool) are detailed in their respective documentation. We recommend consulting those resources for in-depth usage, advanced configurations, and comprehensive examples. This document serves as a high-level introduction, and a deeper dive into individual tool documentation will provide the most practical guidance for your specific infrastructure needs.