Agents

Agents are the workers that execute gather and scrape tasks. They poll the Planekeeper API for pending work, run the task locally, and report results back. You need at least one running agent for jobs to execute.

How agents work

1. Register – on startup, the agent sends a heartbeat to the API server, declaring its capabilities and available credentials.
2. Poll – the agent periodically asks the server for pending tasks (default: every 30 seconds).
3. Execute – when a task is assigned, the agent runs the gather or scrape operation locally.
4. Report – the agent sends the results (or failure details) back to the server.
5. Repeat – the agent continues polling for the next task.

Agent                          API Server
  |                                |
  |-- Heartbeat (capabilities) --> |
  |                                |
  |-- Poll for tasks ------------> |
  |<-- Task assignment ----------- |
  |                                |
  |   (execute task locally)       |
  |                                |
  |-- Report results ------------> |
  |                                |
  |-- Poll for tasks ------------> |
  |<-- No tasks available -------- |
  |                                |
  |   (wait poll interval)         |
  |                                |

Capabilities

During heartbeat, agents declare which job types they can handle:

All capabilities are supported by default.

Task assignment

The server assigns tasks based on:

• Job priority: Jobs are assigned in order of their scheduled run time (oldest first)
• Credential matching: Jobs requiring a specific credential are only assigned to agents that have it
• Distributed locking: Multiple agents can poll simultaneously without claiming the same job

Deploy the client agent

The client agent runs as a Docker container at your site. It connects to the Planekeeper API over HTTPS to poll for tasks and report results.

1. Create an API key

In the Planekeeper UI, navigate to the API Keys page and create a new key. Give it a descriptive name like remote-site-west or prod-datacenter-agent. Copy the key immediately — it is only shown once.

2. Create the compose file

Create a directory for the agent and add the following docker-compose.yml:

name: planekeeper-client

services:
  clientagent:
    image: sqljames/planekeeper-agent:latest
    container_name: planekeeper-client-agent
    restart: unless-stopped
    environment:
      - AGENT_SERVER_URL=${AGENT_SERVER_URL}
      - AGENT_API_KEY=${AGENT_API_KEY}
      - AGENT_POLL_INTERVAL=${AGENT_POLL_INTERVAL:-30}
      - GITHUB_TOKEN=${GITHUB_TOKEN:-}
    volumes:
      - agent-git-cache:/tmp/planekeeper
      # Agent config for private repository credentials
      - ./config.yaml:/etc/planekeeper/config.yaml:ro
      # Uncomment to mount an SSH key file for private repos
      # - ~/.ssh/id_ed25519:/ssh/id_ed25519:ro

volumes:
  agent-git-cache:

3. Create the .env file

In the same directory, create a .env file:

# Required: Planekeeper API endpoint (must include /api/v1/agent)
AGENT_SERVER_URL=https://planekeeper.example.com/api/v1/agent

# Required: The API key you created in step 1
AGENT_API_KEY=pk_xxx

# Optional: Poll interval in seconds (default: 30)
AGENT_POLL_INTERVAL=30

# Optional: GitHub token for higher rate limits (60/hr without, 5000/hr with)
# Create at: https://github.com/settings/tokens
GITHUB_TOKEN=

4. Start the agent

docker compose up -d
docker compose logs -f

You should see a heartbeat message confirming the agent has registered with the server. The agent is also visible on the Dashboard page in the UI.

Configure credentials

Agents can hold named credentials for accessing private Git repositories and container registries. Scrape jobs reference these credentials by name, and the server only assigns credential-requiring jobs to agents that have them.

Credential configuration file

Create a config.yaml file in the same directory as your compose file:

SSH key (file-based)

```yaml title="config.yaml"
agent:
  credentials:
    my-github-ssh:
      type: ssh_key
      private_key_file: /ssh/id_ed25519
      passphrase: ""  # optional
```

Uncomment the SSH volume mount in `docker-compose.yml` and ensure the `private_key_file` path matches the container mount path.

SSH key (inline)

```yaml title="config.yaml"
agent:
  credentials:
    my-github-ssh:
      type: ssh_key
      private_key: |
        -----BEGIN OPENSSH PRIVATE KEY-----
        ...key content...
        -----END OPENSSH PRIVATE KEY-----
      passphrase: ""  # optional
```

HTTPS token

```yaml title="config.yaml"
agent:
  credentials:
    my-github-pat:
      type: https_pat
      token: ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

OCI registry

```yaml title="config.yaml"
agent:
  credentials:
    my-registry:
      type: registry_basic
      username: myuser
      password: dckr_pat_xxxxxxxxxxxx
```

Link credentials to jobs

When creating or editing a scrape job, set the credential name field to the name defined in the config file (for example, my-github-ssh or my-github-pat).

The server checks each agent’s declared credentials during task assignment. Only agents with the matching credential receive the job.

See Private repo access for platform-specific credential examples (GitHub, GitLab, Bitbucket).

Agent health monitoring

Planekeeper tracks agent health through heartbeats and automatic recovery:

Heartbeats

Agents send periodic heartbeats (default: every 30 seconds). The API server records the last heartbeat timestamp for each agent. You can view connected agents and their status on the Dashboard page.

Automatic job recovery

Two mechanisms prevent jobs from getting stuck when agents have problems:

Orphan detection: Every 2 minutes, the server checks for jobs claimed by agents that are no longer registered (no recent heartbeat). These jobs are reset to pending so another agent can pick them up.

Stale job detection: Jobs that have been in “in progress” status for more than one hour are automatically reset to pending. This handles cases where an agent crashes without reporting failure.

Scaling

You can run multiple agents simultaneously. Each agent independently polls for work, and the server uses distributed locking to ensure no two agents claim the same job. Add more agents to increase throughput for large numbers of gather and scrape jobs.

Troubleshooting

Agent cannot connect to the server:

• Verify the AGENT_SERVER_URL includes the full path (https://example.com/api/v1/agent)
• Check that the remote site can reach the server URL (try curl $URL/health)
• Confirm the API key is valid and active

Jobs stay in pending status:

• Check that at least one agent is running and connected (visible on the Dashboard)
• If the job requires a credential, verify the agent has it configured
• Check agent logs for errors during task execution

Agent disconnects frequently:

• Check network stability between the agent and server
• Increase the poll interval if the network is unreliable
• Review agent logs for timeout or connection errors