log_prompt_result
Log an LLM call result to Humanloop. Use this when calling model providers directly (not through Humanloop's proxy) and you want to record the result for observability, evaluation, or feedback. Captures inputs, outputs, token usage, latency, and optional metadata.
manage_tool
Create, update, retrieve, or delete Humanloop tools. Tools represent external functions or capabilities callable by Prompts and Agents. Each tool version is defined by its source code and function schema. Supports tool types including json_schema, python, snippet, and API calls.
manage_logs
List, retrieve, or delete LLM call logs. Logs capture every prompt call or LLM response, including inputs, outputs, latencies, token counts, and costs. Supports filtering by version, date range, and text search. Use this for observability and debugging of your AI applications.
manage_evaluator
Create, update, retrieve, or delete evaluators. Evaluators judge the output of Prompts, Tools, Flows, or other Evaluators. Supports three evaluator types: **Code** (deterministic rules), **AI/LLM** (using foundation models), and **Human** (manual feedback). Returns judgments as booleans, numbers, or selections.
deploy_prompt
Deploy or undeploy a prompt version to a specific environment. Environments control which prompt version is served via the API (e.g. staging, production). Also supports listing prompt versions and viewing current environment deployments.
manage_directory
Create, update, retrieve, list, or delete directories. Directories organize Prompts, Evaluators, Datasets, Flows, and Tools into a hierarchical file structure. Retrieve a directory to see its contents including subdirectories and files.
manage_flow
Create, update, retrieve, or delete flows. Flows are orchestrations of Prompts, Tools, and other code — enabling evaluation and improvement of complete multi-step AI pipelines. Each flow version is identified by its attributes.
run_evaluation
Create and run evaluations, or retrieve evaluation results. Evaluations benchmark different prompt/tool/flow versions against a dataset using specified evaluators. Use this to list evaluations for a file, get evaluation details, or kick off a new evaluation run.
call_prompt
Call a prompt through Humanloop's LLM proxy, which forwards the request to the model provider and automatically logs the result. Supports specifying input variables, messages, and provider API keys. Returns the model's generated output along with usage metadata.
manage_prompt
Create, update, or retrieve prompts with versioned templates. Supports creating new prompts with model configuration and template messages, updating prompt metadata (path, name), and fetching prompt details including version history. Use this to manage your LLM prompt configurations.
manage_dataset
Create, update, retrieve, or delete datasets and their datapoints. Datasets are collections of test cases used for evaluations and fine-tuning. Each datapoint contains inputs, optional messages, and optional target outputs. Supports adding, removing, or replacing datapoints in a dataset.
Manage repositories, issues, and pull requests. Create and configure branches, star repositories, review code, and merge changes. Automate CI/CD workflows with GitHub Actions, manage workflow runs, secrets, and artifacts. Track issues with labels, milestones, and assignees. Search across code, repositories, issues, and users. Manage organizations, teams, and memberships. Create and manage projects, gists, packages, deployments, and environments. Access security alerts including code scanning, secret scanning, and Dependabot alerts. Read and write file contents in repositories. Manage webhooks, notifications, and codespaces.
Manage SharePoint sites, document libraries, lists, and files. Create, read, update, and delete lists and list items with custom columns. Upload, download, move, copy, and version files in document libraries. Search across sites, files, folders, lists, and list items using Microsoft Search. Manage permissions at site, list, and item levels with granular access control. Define and manage content types and site columns. Subscribe to webhooks for list and library change notifications. Retrieve site properties and search for sites across Microsoft 365.
Manage CRM data including Accounts, Contacts, Leads, Opportunities, Cases, and custom objects. Create, read, update, and delete records. Query data using SOQL and search across objects using SOSL. Perform bulk data operations for large-scale imports, exports, and migrations. Execute composite requests to batch multiple operations in a single API call. Access analytics, reports, and dashboards. Manage files and attachments associated with records. Interact with Chatter feeds, posts, and groups for social collaboration. Subscribe to real-time change events via Change Data Capture and Platform Events. Manage org metadata including custom objects, fields, layouts, and workflows. Query data using GraphQL for precise data retrieval across related objects.
Create, read, update, and delete records in Airtable bases and tables. Manage base schemas including creating tables and fields. Filter records using formulas, sort by fields, and scope queries to specific views. Upsert records to find, create, or update in a single call. Upload attachments to records, read and write record comments, list accessible bases, and receive real-time base change events through webhooks.
Manage Git repositories, pull requests, and CI/CD pipelines on Bitbucket Cloud. Create, fork, and configure repositories within workspaces and projects. Create, review, approve, merge, and decline pull requests with inline code comments. Browse source code, list commits, and manage branches and tags. Track issues with the built-in issue tracker. Trigger, monitor, and manage Bitbucket Pipelines. List workspace members, configure repository default reviewers and branch restrictions, create and manage repository webhooks, and search code across repositories.
Deploy, manage, and scale applications on Heroku's cloud platform. Create and configure apps, scale dynos, provision add-ons (databases, caching, etc.), manage configuration variables, build and release code, add custom domains and SSL certificates, manage collaborators and team permissions, configure pipelines for continuous delivery, set up log drains, and sync data with Salesforce via Heroku Connect. Subscribe to webhooks for real-time notifications on app changes, builds, releases, dyno lifecycle events, and more.
Manage LLM prompts, evaluations, and observability for AI applications. Create, version, and deploy prompts with templated variables. Run evaluations using code-based, AI-based, or human evaluators against datasets. Log LLM calls with inputs, outputs, latencies, token counts, and costs. Proxy calls to model providers (OpenAI, Anthropic, Cohere, AWS Bedrock) with automatic logging. Manage datasets of test cases for benchmarking. Configure and trace multi-step agent workflows and flows. Deploy prompt versions to different environments (staging, production). Organize files into directories and attach human feedback to logs.
Common questions about connecting Humanloop to AI agents with Metorial.
