Skip to content

README.md

Latest commit

 

History

History
259 lines (186 loc) · 6.98 KB

README.md

File metadata and controls

259 lines (186 loc) · 6.98 KB
Raw

Jobnik Worker Boilerplate

A production-ready TypeScript boilerplate for building distributed task workers using the MapColonies Jobnik SDK.

Features

  • Type-safe task handling with full TypeScript support for job and stage definitions
  • Built-in observability with distributed tracing, Prometheus metrics, and structured logging
  • Production-ready containerization with multi-stage Docker builds and Helm charts
  • Dependency injection using tsyringe for clean, testable architecture
  • Health checks and graceful shutdown with @godaddy/terminus
  • Comprehensive testing setup with Vitest for unit and integration tests

Prerequisites

  • Node.js >= 24.0.0
  • Connection to a Jobnik Job Manager API instance

Installation

npm install

Configuration

Configure the worker by editing files in the config/ directory:

  • default.json - Base configuration
  • development.json - Development overrides
  • production.json - Production overrides
  • test.json - Test environment settings
  • local.json - Local overrides (not committed to version control)

Set the Jobnik Job Manager API URL via Helm values or environment variables.

Quick Start

Development Mode

npm run start:dev

Development mode enables offline config mode and source map support for better debugging.

Production Build

npm run build
npm start

Running Tests

# Run all tests with coverage
npm test

# Watch mode for development
npm run test:watch

Integration tests

npm run test:integration needs an S3-compatible server and a Redis server. By default it starts an S3-compatible and a Redis testcontainer automatically. The images are pinned to the releases deployed in our environments; bump them in tests/integration/helpers/s3Container.ts and tests/integration/helpers/redisContainer.ts when those environments move.

To run against already-running servers instead, set the TEST_* variables:

Variable Default Purpose
TEST_S3_ENDPOINT (unset) Point the suite at an existing S3 server. Unset means start a container.
TEST_S3_ACCESS_KEY minioadmin Access key for that server.
TEST_S3_SECRET_KEY minioadmin Secret key for that server.
TEST_REDIS_HOST (unset) Point the suite at an existing Redis. Unset means start a container.
TEST_REDIS_PORT 6379 Port for that server.

The suite creates and deletes buckets on whichever S3 server it is given, and calls FLUSHDB on whichever Redis it is given. Never point TEST_S3_ENDPOINT or TEST_REDIS_HOST at a shared or deployed environment, and beware of leaving them exported in a shell profile. Each run prints which mode it selected and against which server.

Customizing the Boilerplate

This boilerplate includes example "logistics" code to demonstrate task handling. Follow these steps to adapt it to your use case:

1. Remove Example Code

Delete the example logistics implementation:

rm -rf src/logistics src/seeder.ts tests/logistics.spec.ts

Remove the seeder call from src/index.ts:

// REMOVE THESE LINES:
const sdk = container.resolve<LogisticsSDK>(SERVICES.JOBNIK_SDK);
await seedData(sdk.getProducer());

2. Define Your Job and Stage Types

Create a new types file (e.g., src/yourDomain/types.ts):

import type { IJobnikSDK } from '@map-colonies/jobnik-sdk';

export interface YourJobTypes {
  jobType1: {
    data: { /* your job data schema */ };
    userMetadata: { /* your job metadata */ };
  };
}

export interface YourStageTypes {
  stage1: {
    data: { /* stage data schema */ };
    userMetadata: { /* stage metadata */ };
    task: { 
      data: { /* task data schema */ }; 
      userMetadata: { /* task metadata */ } 
    };
  };
}

export type YourSDK = IJobnikSDK<YourJobTypes, YourStageTypes>;

3. Create Task Handler

Create your manager (e.g., src/yourDomain/manager.ts):

import { injectable } from 'tsyringe';
import type { Task, TaskHandlerContext } from '@map-colonies/jobnik-sdk';

@injectable()
export class YourManager {
  public async handleYourTask(
    task: Task<YourStageTypes['stage1']['task']>,
    context: TaskHandlerContext<YourJobTypes, YourStageTypes, 'jobType1', 'stage1'>
  ): Promise<void> {
    context.logger.info({ msg: 'Processing task', taskId: task.id });
    
    // Your task processing logic here
    
    await context.updateStageUserMetadata({ /* updated metadata */ });
  }
}

4. Update Worker Configuration

Modify src/worker.ts to use your new types and handler:

import { YourManager } from './yourDomain/manager';
import type { YourSDK } from './yourDomain/types';

export const workerBuilder: FactoryFunction<IWorker> = (container: DependencyContainer) => {
  const sdk = container.resolve<YourSDK>(SERVICES.JOBNIK_SDK);
  const manager = container.resolve(YourManager);
  
  const worker = sdk.createWorker<'jobType1', 'stage1'>(
    'stage1',
    manager.handleYourTask.bind(manager),
    config.get('jobnik.worker')
  );
  
  return worker;
};

5. Rename Throughout the Project

Update references to cleaner in:

  • package.json - name, description, author
  • helm/Chart.yaml - name, description
  • helm/values.yaml - mclabels, configManagement.name, image.repository
  • helm/templates/_helpers.tpl - all template definitions
  • helm/templates/*.yaml - review all template files for hardcoded references

6. Update Package Metadata

Edit package.json:

{
  "name": "your-worker-name",
  "description": "Your worker description",
  "author": "Your Team"
}

Observability

Metrics

Prometheus metrics are exposed on the /metrics endpoint (default port 8080). Key metrics include:

  • Worker task processing duration
  • Task success/failure rates
  • Active task count
  • Custom application metrics

Tracing

Distributed tracing can be enabled in config/default.json:

{
  "telemetry": {
    "tracing": {
      "isEnabled": true,
      "url": "http://your-otlp-collector:4318/v1/trace"
    }
  }
}

Logging

Structured JSON logging is provided by @map-colonies/js-logger. Configure log level:

{
  "telemetry": {
    "logger": {
      "level": "info",
      "prettyPrint": false
    }
  }
}

Deployment

Kubernetes/Helm

Deploy using Helm:

helm install your-worker-name ./helm

Documentation