README_M2M_AUTH.md

M2M Token Authentication Setup Guide

This guide explains how to set up Machine-to-Machine (M2M) authentication between the bot services (CandidateRepresenter.Bot and JobApplier.Bot) and the JobTracker API using Clerk.

Overview

The JobTracker API is now protected with Clerk M2M token authentication. Services like CandidateRepresenter.Bot and JobApplier.Bot must authenticate using M2M tokens to access the API.

Architecture

┌─────────────────────────────────────┐
│ CandidateRepresenter.Bot (Python)   │
│                                     │
│ 1. Load CLERK_MACHINE_SECRET_KEY    │
│ 2. On startup: Create M2M token     │
│ 3. Cache token for session (1 hour) │
│ 4. Attach to all API requests       │
└────────────┬────────────────────────┘
             │ HTTP Request
             │ Authorization: Bearer mt_xxx
             ▼
┌─────────────────────────────────────┐
│ JobTracker API (.NET)               │
│                                     │
│ 1. Extract Bearer token             │
│ 2. If starts with "mt_":            │
│    - Call Clerk verify endpoint     │
│    - Validate scopes/expiration     │
│ 3. If standard JWT:                 │
│    - Use existing JWKS validation   │
│ 4. Proceed with request             │
└─────────────────────────────────────┘
             ▲
             │ HTTP Request
             │ Authorization: Bearer mt_xxx
┌────────────┴────────────────────────┐
│ JobApplier.Bot (Python)             │
│                                     │
│ 1. Load CLERK_MACHINE_SECRET_KEY    │
│ 2. On startup: Create M2M token     │
│ 3. Cache token for session (1 hour) │
│ 4. Attach to all API requests       │
└─────────────────────────────────────┘

Setup Instructions

1. Configure Machines in Clerk Dashboard

1. Navigate to the Clerk Dashboard Machines page

2. Create CandidateRepresenter.Bot machine:

   • Click "Add machine"
   • Name: CandidateRepresenter.Bot
   • Description: "Python bot for candidate question answering"
   • Click "Create"
   • Important: Copy and save the machine secret key (starts with ak_)
   • You'll need this for the Bot's .env file

3. Create JobApplier.Bot machine:

   • Click "Add machine"
   • Name: JobApplier.Bot
   • Description: "Python bot for job application automation"
   • Click "Create"
   • Important: Copy and save the machine secret key (starts with ak_)
   • You'll need this for the Bot's .env file

4. Create JobTracker.API machine:

   • Click "Add machine"
   • Name: JobTracker.API
   • Description: ".NET Web API for job tracking"
   • In the "Scopes" section, select both CandidateRepresenter.Bot and JobApplier.Bot
   • Click "Create"
   • Important: Copy and save the machine secret key (starts with ak_)
   • You'll need this for the API's appsettings.json file

5. Configure bi-directional communication:

   • Find CandidateRepresenter.Bot in the list

   • Click "Edit machine"

   • In the "Scopes" section, select JobTracker.API

   • Click "Update"

   • Find JobApplier.Bot in the list

   • Click "Edit machine"

   • In the "Scopes" section, select JobTracker.API

   • Click "Update"

All machines can now create tokens to authenticate with each other.

2. Configure CandidateRepresenter.Bot

1. Update environment variables:

cd CandidateRepresenter.Bot
cp .env.example .env

2. Edit .env file:

# Clerk M2M Authentication
CLERK_MACHINE_SECRET_KEY=ak_your_actual_secret_key_here
CLERK_API_BASE_URL=https://api.clerk.com/v1

# JobTracker API Configuration
JOBTRACKER_API_URL=http://localhost:5000

3. The JobTrackerClient will automatically:
   • Create M2M tokens on initialization
   • Cache tokens for 1 hour (with 5-minute safety buffer)
   • Refresh tokens automatically before expiration
   • Retry with fresh token on 401 responses

3. Configure JobApplier.Bot

1. Update environment variables:

cd JobApplier.Bot
cp .env.example .env

2. Edit .env file:

# Clerk M2M Authentication
CLERK_MACHINE_SECRET_KEY=ak_your_actual_secret_key_here
CLERK_API_BASE_URL=https://api.clerk.com/v1

# JobTracker API Configuration
JOBTRACKER_API_URL=http://localhost:5138

3. The JobTrackerClient will automatically:
   • Create M2M tokens on initialization
   • Cache tokens for 1 hour (with 5-minute safety buffer)
   • Refresh tokens automatically before expiration
   • Retry with fresh token on 401 responses

4. Configure JobTracker API

The API needs its own machine secret key to authenticate with Clerk when verifying tokens.

1. Copy the example configuration file:

cd JobTracker
cp appsettings.Development.json.example appsettings.Development.json

2. Edit appsettings.Development.json with your actual secret key:

{
  "Clerk": {
    "M2M": {
      "MachineSecretKey": "ak_your_actual_jobtracker_api_secret_key_here"
    }
  }
}

3. Important Security Note:
   • Never commit appsettings.Development.json to source control (already in .gitignore)
   • The example file (appsettings.Development.json.example) is tracked in git as a template
   • In production, use environment variables or Azure Key Vault for secrets
   • The base appsettings.json contains no secrets and is safe to commit

How it works:

• When the Bot sends a request with an M2M token (mt_xxx)
• The API uses its own machine secret key to authenticate with Clerk
• Clerk verifies the incoming token and returns validation results
• The API allows or denies the request based on verification

Usage

Python (CandidateRepresenter.Bot & JobApplier.Bot)

The authentication is automatic when using the JobTrackerClient:

from src.clients.jobtracker_client import JobTrackerClient

# M2M authentication is enabled by default
async with JobTrackerClient() as client:
    # Automatically adds M2M token to all requests
    application = await client.get_job_application(application_id)
    user = await client.get_user(user_id)
    
    # For JobApplier.Bot: Additional operations
    await client.update_status(application_id, status=2)  # Mark as Submitted
    await client.download_resume(resume_id, save_path)

To disable M2M authentication (for testing):

async with JobTrackerClient(enable_m2m_auth=False) as client:
    # No authentication headers
    pass

.NET (JobTracker API)

The API automatically handles both user JWT tokens and M2M tokens:

• User JWTs (standard Clerk tokens): Validated via JWKS
• M2M tokens (start with mt_): Validated via Clerk API
• Both authentication methods work simultaneously

Monitoring & Logging

Token Creation Costs

Clerk charges for M2M tokens:

• Token creation: $0.001 per token
• Token verification: $0.0001 per verification

With 1-hour token lifetime and session-based caching:

• CandidateRepresenter.Bot: ~$0.024/day (1 token/hour × 24 hours)
• JobApplier.Bot: ~$0.024/day (1 token/hour × 24 hours)
• Total Bot Token Creation: ~$0.048/day
• API verifications: Variable based on request volume

Logging

Python side (Both Bots):

INFO: M2M authentication enabled for JobTracker API
INFO: Creating new M2M token (lifetime: 3600s)
INFO: M2M token created successfully (expires at 2025-01-07 12:43:00, will refresh at 12:38:00)
DEBUG: Using cached M2M token (expires in 0:54:23)
DEBUG: Added M2M token to request headers

API side (JobTracker):

DEBUG: Detected M2M token, attempting verification
DEBUG: Verifying M2M token with Clerk API
INFO: M2M token verified successfully. Subject: mch_xxx, Expired: False, Revoked: False
INFO: M2M authentication successful. Machine: mch_xxx, Scopes: mch_yyy

Troubleshooting

401 Unauthorized Errors

Problem: API returns 401 even with M2M token

Solutions:

1. Verify machine secret key is correct in .env
2. Check that machines are configured with proper scopes in Clerk Dashboard
3. Ensure token hasn't been revoked in Clerk Dashboard
4. Check API logs for specific error messages

Token Refresh Issues

Problem: Bot fails to refresh expired tokens

Solution: The client automatically refreshes tokens 5 minutes before expiration. If issues persist:

1. Check network connectivity to api.clerk.com
2. Verify Clerk API is accessible
3. Check logs for specific error messages

M2M Token Not Being Used

Problem: Bot makes requests without authentication

Solutions:

1. Verify CLERK_MACHINE_SECRET_KEY is set in .env
2. Check that enable_m2m_auth=True (default)
3. Look for warning logs about missing configuration

Security Best Practices

1. Never commit machine secret keys

   • Always use .env files (already in .gitignore)
   • Rotate keys periodically in Clerk Dashboard

2. Monitor token usage

   • Review logs regularly for authentication failures
   • Set up alerts for unusual token creation patterns

3. Scope management

   • Only grant necessary scopes between machines
   • Review and update scopes when adding new services

4. Token lifetime

   • Default 1-hour lifetime balances security and cost
   • Adjust via ClerkM2MClient(token_lifetime_seconds=X)

Testing

Manual Testing with Swagger

1. Start JobTracker API
2. Navigate to http://localhost:5000/swagger
3. Click "Authorize"
4. Enter an M2M token in the format: mt_xxx
5. Try API endpoints

Integration Testing

Test files verify M2M authentication:

• CandidateRepresenter.Bot/tests/test_jobtracker_client.py
• Mock Clerk API responses for token creation/verification

Migration Guide for Other Services

To add M2M authentication to other services:

1. Create machine in Clerk Dashboard
2. Add configuration (similar to CandidateRepresenter.Bot)
3. Update HTTP client to include M2M token in Authorization header
4. Test thoroughly before deploying

See CandidateRepresenter.Bot/src/clients/clerk_m2m_client.py as reference implementation.

References

• Clerk M2M Documentation
• Clerk Dashboard
• JobTracker API: /swagger endpoint