Development of AI Applications with Effect

Integration with large language models (LLMs) has become an essential part of modern application development. Whether you're creating content, analyzing data, or developing user interaction interfaces, adding AI-based capabilities has the potential to both extend your product's functionality and enhance the user experience.

However, successfully integrating LLM-based capabilities into an application can prove to be quite challenging. Developers must navigate a variety of potential failures: network errors, provider outages, rate limits, and much more—these all need to be handled to ensure the application's stability and responsiveness for the end user. Additionally, differences between language model provider APIs can force developers to write fragile "glue code," which may later become a significant source of technical debt.

Today, we'll take a look at AI integration packages from Effect—a set of libraries designed to simplify working with LLMs, ensuring flexibility and independence from any specific provider.

Why Effect for AI?

Effect's AI packages provide simple, compositional building blocks for modeling interactions with LLMs in a safe, declarative, and modular style. With them, you can:

🔌 Write provider-agnostic business logic

Describe your interaction with an LLM once, and then simply plug in the necessary provider. This allows you to switch between any supported providers without affecting the business logic.

🧪 Test interactions with LLMs

Conduct testing by providing mock implementations of services to ensure that the AI-dependent logic works as expected.

🧵 Use structured concurrency

Run parallel LLM calls, cancel outdated requests, implement streaming of partial results, or organize "races" between multiple providers—all safely managed by Effect's structured concurrency model.

🔍 Gain advanced observability

Instrument LLM interactions with built-in tracing, logging, and metrics to identify performance bottlenecks or production failures.

Understanding the Package Ecosystem

The Effect AI ecosystem consists of several specialized packages, each serving its own purpose:

• @effect/ai: the base package that defines provider-agnostic services and abstractions for interacting with LLMs.

• @effect/ai-openai: specific implementations of AI services based on the OpenAI API.

• @effect/ai-anthropic: specific implementations of AI services based on the Anthropic API.

This architecture allows you to describe interactions with LLMs using provider-independent services and then plug in a specific implementation when running the program.

Key Concepts

Provider-Agnostic Programming

The core philosophy behind Effect's AI integrations is provider-agnostic programming.

Instead of hardcoding API calls to a particular LLM provider, you describe the interaction using universal services from the base @effect/ai package.

import { Completions } from "@effect/ai"
import { Effect } from "effect"

// Define a provider-agnostic AI interaction
const generateDadJoke = Effect.gen(function*() {
  // Get the Completions service from the Effect environment
  const completions = yield* Completions.Completions

  // Use the service to generate text
  const response = yield* completions.create("Generate a dad joke")

  // Return the response
  return response
})

This separation of concerns is fundamental to the Effect approach to LLM interaction.

AiModel Abstraction

To bridge the gap between provider-independent business logic and specific LLM providers, Effect introduces the AiModel abstraction.

AiModel represents a particular LLM from a certain provider, which can fulfill service requirements such as Completions or Embeddings.

import { OpenAiCompletions } from "@effect/ai-openai"
import { Effect } from "effect"

import { Completions } from "@effect/ai"

// Define a provider-agnostic AI interaction
const generateDadJoke = Effect.gen(function*() {
  // Get the Completions service from the Effect environment
  const completions = yield* Completions.Completions

  // Use the service to generate text
  const response = yield* completions.create("Generate a dad joke")

  // Return the response
  return response
})

// Create an AiModel for OpenAI's GPT-4o
const Gpt4o = OpenAiCompletions.model("gpt-4o")

// Use the model to provide the Completions service to our program
const main = Effect.gen(function*() {
  // Build the AiModel into a Provider
  const gpt4o = yield* Gpt4o

  // Provide the implementation to our generateDadJoke program
  const response = yield* gpt4o.provide(generateDadJoke)

  console.log(response.text)
})

The advantages of this approach:

• Reusability: you can use the same model for multiple operations

• Flexibility: easily switch between providers or models as needed

• Abstraction: encapsulate AI logic into services that hide implementation details.

End-to-End Example

Let's look at a complete example of setting up LLM interaction using Effect

import { OpenAiClient, OpenAiCompletions } from "@effect/ai-openai"
import { Completions } from "@effect/ai"
import { NodeHttpClient } from "@effect/platform-node"
import { Config, Effect, Layer } from "effect"

// 1. Define our provider-agnostic AI interaction
const generateDadJoke = Effect.gen(function*() {
  const completions = yield* Completions.Completions
  const response = yield* completions.create("Generate a dad joke")
  return response
})

// 2. Create an AiModel for a specific provider and model
const Gpt4o = OpenAiCompletions.model("gpt-4o")

// 3. Create a program that uses the model
const main = Effect.gen(function*() {
  const gpt4o = yield* Gpt4o
  const response = yield* gpt4o.provide(generateDadJoke)
  console.log(response.text)
})

// 4. Create a Layer that provides the OpenAI client
const OpenAi = OpenAiClient.layerConfig({
  apiKey: Config.redacted("OPENAI_API_KEY")
})

// 5. Provide an HTTP client implementation
const OpenAiWithHttp = Layer.provide(OpenAi, NodeHttpClient.layerUndici)

// 6. Run the program with the provided dependencies
main.pipe(
  Effect.provide(OpenAiWithHttp),
  Effect.runPromise
)

The example above demonstrates the main steps:

1. Define AI interactions, independent of the provider.

2. Create an AiModel for a particular provider and model.

3. Develop a program that uses this model.

4. Create a layer that provides the OpenAI client.

5. Provide an HTTP client.

6. Run the program with the required dependencies.

Advanced Features

Error Handling

One of Effect’s strengths is its reliable error handling, which is especially valuable when working with LLMs, where potential failure scenarios can be complex and varied. With Effect, errors are typed and can be handled explicitly.

For example, if a joke generator program needs to be rewritten so it can fail with RateLimitError or InvalidInputError, the relevant error handling logic can be implemented.

import { AiResponse, AiRole } from "@effect/ai"
import { Effect } from "effect"

import { Completions } from "@effect/ai"
import { Data } from "effect"

class RateLimitError extends Data.TaggedError("RateLimitError") {}
class InvalidInputError extends Data.TaggedError("InvalidInputError") {}

declare const generateDadJoke: Effect.Effect<
  AiResponse.AiResponse,
  RateLimitError | InvalidInputError,
  Completions.Completions
>

const withErrorHandling = generateDadJoke.pipe(
  Effect.catchTags({
    RateLimitError: (error) =>
      Effect.logError("Rate limited, retrying in a moment").pipe(
        Effect.delay("1 seconds"),
        Effect.andThen(generateDadJoke)
      ),
    InvalidInputError: (error) =>
      Effect.succeed(AiResponse.AiResponse.fromText({
        role: AiRole.model,
        content: "I couldn't generate a joke right now."
      }))
  })
)

Structured execution plans

For more complex scenarios that require high reliability when working with multiple providers, Effect provides a powerful abstraction: AiPlan.

AiPlan lets you create structured execution plans for LLM interactions, with built-in retry logic, fallback strategies, and error handling.

import { AiPlan } from "@effect/ai"
import { OpenAiCompletions } from "@effect/ai-openai"
import { AnthropicCompletions } from "@effect/ai-anthropic"
import { Data, Effect, Schedule } from "effect"

import { Completions } from "@effect/ai"

const generateDadJoke = Effect.gen(function*() {
  const completions = yield* Completions.Completions
  const response = yield* completions.create("Generate a dad joke")
  return response
})

// Define domain-specific error types
class NetworkError extends Data.TaggedError("NetworkError") {}
class ProviderOutage extends Data.TaggedError("ProviderOutage") {}

// Build a resilient plan that:
// - Attempts to use OpenAI's `"gpt-4o"` model up to 3 times
// - Waits with an exponential backoff between attempts
// - Only re-attempts the call to OpenAI if the error is a `NetworkError`
// - Falls back to using Anthropic otherwise
const DadJokePlan = AiPlan.fromModel(OpenAiCompletions.model("gpt-4o"), {
  attempts: 3,
  schedule: Schedule.exponential("100 millis"),
  while: (error: NetworkError | ProviderOutage) =>
    error._tag === "NetworkError"
}).pipe(
  AiPlan.withFallback({
    model: AnthropicCompletions.model("claude-3-7-sonnet-latest"),
  })
)

// Use the plan just like an AiModel
const main = Effect.gen(function*() {
  const plan = yield* DadJokePlan
  const response = yield* plan.provide(generateDadJoke)
})

With AiPlan you can:

• Create complex retry policies with customizable exponential backoff strategies.

• Define fallback chains between multiple providers.

• Specify which error types should trigger retries and which should trigger a fallback.

This is especially valuable for production systems where reliability is critical, since it allows the use of multiple LLM providers as backups, keeping business logic independent of any particular provider.

Concurrency control

Effect’s structured concurrency model makes it easier to manage parallel requests to LLMs:

import { Effect } from "effect"

import { Completions } from "@effect/ai"

const generateDadJoke = Effect.gen(function*() {
  const completions = yield* Completions.Completions
  const response = yield* completions.create("Generate a dad joke")
  return response
})

// Generate multiple jokes concurrently
const concurrentDadJokes = Effect.all([
  generateDadJoke,
  generateDadJoke,
  generateDadJoke
], { concurrency: 2 }) // Limit to 2 concurrent requests

Streaming responses

Effect's AI integrations support streaming responses using the Stream type:

import { Completions } from "@effect/ai"
import { Effect, Stream } from "effect"

const streamingJoke = Effect.gen(function*() {
  const completions = yield* Completions.Completions

  // Create a streaming response
  const stream = completions.stream("Tell me a long dad joke")

  // Process each chunk as it arrives
  return yield* stream.pipe(
    Stream.runForEach(chunk =>
      Effect.sync(() => {
        process.stdout.write(chunk.text)
      })
    )
  )
})

Conclusion

Whether you are building an intelligent agent, an interactive chat, or a system using LLM for background tasks, Effect's AI packages provide all the necessary tools and more. Our provider-independent approach ensures that your code remains adaptable as the AI landscape evolves.

Ready to try Effect for your next AI application? Check out the “Getting Started” guide.

Effect AI integration packages are in experimental/alpha stages, but we highly encourage you to try them out and provide feedback to help us improve and expand their capabilities.

We look forward to seeing your projects! Check out the full documentation for a deeper dive and join our community to share your experience and get support.