Skip to content

OpenAI Setup Guide

What You'll Learn

How to connect your OpenAI account to RunForge to track costs, performance, and usage of GPT models.

Why Choose OpenAI?

  • Industry standard: Most widely used AI provider
  • High quality: GPT-4 and GPT-4o models offer excellent performance
  • Reliable service: Strong uptime and consistent performance
  • Wide model selection: From fast/cheap GPT-4o-mini to powerful GPT-4

Before You Start

Step 1: Create Your OpenAI API Key

Sign In to OpenAI Platform

  1. Go to platform.openai.com
  2. Sign in with your OpenAI account
  3. If you don't have an account, click Sign up and create one

Generate an API Key

  1. Click your profile in the top-right corner
  2. Select "View API keys" from the dropdown
  3. Click "Create new secret key"
  4. Give it a name like "RunForge Production" or "My App Key"
  5. Copy the key immediately - you won't be able to see it again!
  6. Store it safely - keep it in a secure password manager

Important Security Note

⚠️ Never share your API key or put it directly in your code. Always use environment variables or secure configuration files.

Why Set Limits?

Prevent unexpected charges if something goes wrong with your application or if your API key gets compromised.

Configure Usage Limits

  1. Go to "Billing" in your OpenAI dashboard
  2. Click "Usage limits"
  3. Set a monthly limit (start with $50-100 for testing)
  4. Set up email notifications at 75% and 90% of your limit
  5. Save your settings
  • Testing/Development: $20-50/month
  • Small Production App: $100-300/month
  • Medium Production App: $500-1000/month
  • Large Production App: Set based on your budget

Step 3: Add OpenAI Key to RunForge

  1. Open RunForge and sign in
  2. Select your project from the project dropdown
  3. Go to Settings → API Keys

Add OpenAI Configuration

  1. Click "Add API Key" or "Add Provider Key"
  2. Select "OpenAI" from the provider dropdown
  3. Fill out the form:
  4. Key Name: "Production OpenAI Key" (or whatever makes sense)
  5. API Key: Paste your OpenAI API key here
  6. Description: "Main OpenAI key for production chatbot" (optional but helpful)
  7. Test the connection - RunForge will verify the key works
  8. Save the configuration

Verify Everything Works

You should see: - ✅ Connection successful message - Key is active status indicator - Usage limits (if configured) showing correctly

Step 4: Configure Your Application

TypeScript/JavaScript Setup

Install the required SDKs: 

npm install openai @runforge/sdk

Set up environment variables (in your .env file): 

# OpenAI Configuration
OPENAI_API_KEY=your-openai-key-here

# RunForge Configuration  
RUNFORGE_API_KEY=your-runforge-ingest-key-here
RUNFORGE_PROJECT_ID=your-project-id-here
RUNFORGE_ENDPOINT=http://localhost:3000/api/ingest

Basic usage example: 

import OpenAI from 'openai';
import { RunForge } from '@runforge/sdk';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY!
});

const runforge = new RunForge({
  apiKey: process.env.RUNFORGE_API_KEY!,
  projectId: process.env.RUNFORGE_PROJECT_ID!,
  endpoint: process.env.RUNFORGE_ENDPOINT!
});

// Make a tracked OpenAI call
async function generateResponse(userMessage: string) {
  const result = await runforge.track(
    { 
      experiment: 'chatbot-v1',
      user_id: 'user-123' // optional metadata
    },
    () => openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [
        { role: 'system', content: 'You are a helpful assistant.' },
        { role: 'user', content: userMessage }
      ],
      temperature: 0.7,
      max_tokens: 500
    })
  );

  return result.choices[0].message.content;
}

Python Setup

Install the required SDKs: 

pip install openai runforge

Set up environment variables: 

# In your .env file or environment
export OPENAI_API_KEY=your-openai-key-here
export RUNFORGE_API_KEY=your-runforge-ingest-key-here  
export RUNFORGE_PROJECT_ID=your-project-id-here
export RUNFORGE_ENDPOINT=http://localhost:3000/api/ingest

Basic usage example: 

import os
from openai import OpenAI
from runforge import RunForge

client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])

rf = RunForge(
    api_key=os.environ['RUNFORGE_API_KEY'],
    project_id=os.environ['RUNFORGE_PROJECT_ID'],
    endpoint=os.environ['RUNFORGE_ENDPOINT']
)

def generate_response(user_message):
    result = rf.track(
        {
            "experiment": "chatbot-v1", 
            "user_id": "user-123"  # optional metadata
        },
        lambda: client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": user_message}
            ],
            temperature=0.7,
            max_tokens=500
        )
    )

    return result.choices[0].message.content

Step 5: Test Your Setup

Make Your First Tracked Call

Run your application and make an AI request. You should see:

In your application: - Normal AI response from OpenAI - No errors or warnings

In RunForge dashboard (within 30 seconds): - New request appears in your dashboard - Cost information shows up - Response time is recorded - Success status is logged

Troubleshooting First Test

If you don't see data in RunForge: 1. Check your RunForge ingest key is correct 2. Verify your project ID matches 3. Make sure your RunForge endpoint URL is accessible 4. Check browser developer tools for any error messages

If you get OpenAI errors: 1. Verify your OpenAI API key is correct 2. Check you have sufficient credit balance 3. Ensure you haven't hit rate limits 4. Confirm your OpenAI account is in good standing

OpenAI Model Guide

  • Cost: ~$0.15/1M input tokens, ~$0.60/1M output tokens
  • Speed: Very fast (~1 second average)
  • Use for: Chatbots, content generation, most applications
  • Quality: Very good for most tasks

GPT-4o

  • Cost: ~$2.50/1M input tokens, ~$10/1M output tokens
  • Speed: Fast (~2 seconds average)
  • Use for: Complex reasoning, high-quality content creation
  • Quality: Excellent, good balance of cost and performance

GPT-4-turbo

  • Cost: ~$10/1M input tokens, ~$30/1M output tokens
  • Speed: Moderate (~3 seconds average)
  • Use for: Complex analysis, creative writing, difficult tasks
  • Quality: Top tier, but expensive

Choosing the Right Model

Start with GPT-4o-mini for: - Chatbots and conversational AI - Content summarization
- Simple Q&A systems - High-volume applications

Upgrade to GPT-4o when you need: - Better reasoning capabilities - More creative or nuanced responses - Complex problem-solving - Higher quality is worth the extra cost

Use GPT-4-turbo only when: - Quality is critical and cost is less important - Complex analysis or reasoning is required - You've tested and confirmed it's significantly better than cheaper options

Cost Optimization Tips

Reduce Token Usage

  • Shorter prompts: Be concise but clear in your instructions
  • Limit max_tokens: Set reasonable limits based on your needs
  • Smart truncation: Cut off irrelevant context

Choose Cost-Effective Models

  • Test cheaper models first: GPT-4o-mini often works as well as GPT-4
  • Use different models for different tasks: Simple tasks don't need expensive models
  • Monitor cost per task: Track which operations are most expensive

Optimize Prompt Strategy

  • Few-shot prompting: Give examples instead of long instructions
  • System prompts: Use system messages for consistent instructions
  • Template prompts: Reuse well-tested prompts

Batch Processing

  • Group similar requests: Process multiple items in one API call when possible
  • Cache results: Don't re-generate identical content
  • Queue non-urgent requests: Process during off-peak hours if rate limits are cheaper

Monitoring and Alerts

Key Metrics to Watch

  • Daily/monthly spend: Track against your budget
  • Cost per successful request: Optimize for efficiency
  • Token usage trends: Identify spikes or inefficiencies
  • Error rates: High errors waste money on failed requests
  1. Budget alert: 80% of monthly OpenAI spend
  2. Usage spike: 3x normal daily usage
  3. High error rate: More than 5% failures
  4. Expensive requests: Individual requests over $1

Common Issues and Solutions

"Exceeded your current quota" Error

Problem: You've hit your OpenAI usage limit Solutions: 1. Check your usage limit in OpenAI dashboard 2. Increase your usage limit if needed 3. Wait until next billing cycle if you're at your limit 4. Add more credit to your OpenAI account

High Costs

Problem: Your OpenAI bill is higher than expected Investigation: 1. Check RunForge dashboard for usage patterns 2. Look for unusually long prompts or responses 3. Verify you're using the right model (not accidentally using GPT-4 instead of GPT-4o-mini) 4. Check for runaway processes or bugs

Slow Response Times

Problem: OpenAI requests are taking too long Solutions: 1. Switch to a faster model (GPT-4o-mini is usually fastest) 2. Reduce max_tokens to get shorter responses faster 3. Check OpenAI status page for service issues 4. Consider using streaming for long responses

Authentication Errors

Problem: "Invalid API key" or authentication failures Solutions: 1. Double-check your API key is copied correctly 2. Ensure the key hasn't been deactivated in OpenAI dashboard
3. Verify you're using the right environment variables 4. Try generating a new API key

Advanced Configuration

Streaming Responses

For better user experience with long responses: 

const stream = await openai.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: messages,
  stream: true,
  stream_options: { include_usage: true } // Important for cost tracking
});

// Handle stream and track with RunForge when complete

Function Calling

Enable your AI to call external functions: 

const result = await runforge.track(
  { experiment: 'function-calling' },
  () => openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: messages,
    functions: [
      {
        name: 'get_weather',
        description: 'Get current weather',
        parameters: {
          type: 'object',
          properties: {
            location: { type: 'string' }
          }
        }
      }
    ],
    function_call: 'auto'
  })
);

Custom Instructions

Set up consistent behavior across all requests: 

const systemPrompt = `You are a professional customer service assistant. 
- Always be polite and helpful
- Keep responses under 200 words  
- If you can't help, offer to escalate to a human
- Use a friendly but professional tone`;

const result = await openai.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [
    { role: 'system', content: systemPrompt },
    { role: 'user', content: userMessage }
  ]
});

Next Steps