Jobs, cache and health

Queue jobs

Walk whole lists, import or sync, off the request, retried on failure.

Cached answers

Read the site once every few minutes, and keep serving if it’s down.

A readiness check

/health/ready fails when the site doesn’t answer.

Queue jobs

A request should answer fast. Walking every page of a list, importing a batch, or a FetchskiAI call that runs for minutes belongs in a job: @adonisjs/queue runs it in a worker and retries it when it fails.

@adonisjs/queue is still marked experimental by AdonisJS: its API may change between minor releases, so read its changelog before you update it.

Add the queue

Terminal

node ace add @adonisjs/queue

Pick database to keep jobs in the database your app already has (or redis). With database, it writes a migration for the jobs tables: node ace migration:run.

Write the job

Terminal

node ace make:job sync_blog_posts

A job takes the client in its constructor, like a controller:

app/jobs/sync_blog_posts.ts

import { inject } from '@adonisjs/core'
import logger from '@adonisjs/core/services/logger'
import { Job } from '@adonisjs/queue'
import type { JobOptions } from '@adonisjs/queue/types'
import { SiteClient } from '@fetchski/attract-cms-sdk'

interface SyncBlogPostsPayload {
  perPage: number
}

@inject()
export default class SyncBlogPosts extends Job<SyncBlogPostsPayload> {
  static options: JobOptions = {
    queue: 'default',
    maxRetries: 3,
  }

  constructor(protected site: SiteClient) {
    super()
  }

  async execute() {
    let count = 0
    for await (const post of this.site
      .blogPosts()
      .listBlogsAll({ perPage: this.payload.perPage })) {
      count++
      logger.debug({ id: post.id }, 'blog post')
    }
    logger.info({ count }, 'Synced the blog posts')
  }
}

listBlogsAll() fetches one page at a time as the loop asks (Paging); put your own work where the logger.debug is.

Dispatch it, or schedule it

import SyncBlogPosts from '#jobs/sync_blog_posts'

await SyncBlogPosts.dispatch({ perPage: 50 })

import app from '@adonisjs/core/services/app'
import SyncBlogPosts from '#jobs/sync_blog_posts'

// Not while `node ace codegen` warms the app up: the queue isn't started then
if (app.getMode() !== 'warmup') {
  await SyncBlogPosts.schedule({ perPage: 50 }).id('sync-blog-posts').every('1h')
}

node ace add registered start/scheduler.ts for you. .cron('0 6 * * *') instead of .every() runs it at a set time. Keep the warmup check: node ace codegen loads this file without starting the queue, and a schedule made then throws and stops it.

Run the worker

Terminal

node ace queue:work

Jobs and schedules run only while a worker does: keep one running beside the web server in production.

Retries: the SDK's and the queue's

The SDK retries a call that is safe to repeat (a read with no answer, a rate limit) before it throws. The queue’s maxRetries reruns the whole job after that. A write that timed out may have landed, so make a job that writes safe to run twice: check before you create.

Long FetchskiAI calls

AI routes carry their own timeout (up to 615 seconds), which the SDK applies for you. A job has no timeout unless you give it one (its timeout option, or the worker’s): keep any you set above that.

Testing a job

QUEUE_DRIVER=sync in .env.test runs a dispatched job at once, with the fakeAttract() fake: Testing → A queue job.

Cache the answers

Content that changes a few times a day needn’t be read on every request. With @adonisjs/cache, read it once every ten minutes, and keep serving the last answer for six hours if the site is down:

Add the cache

Terminal

node ace add @adonisjs/cache

It asks for a store (Redis, file, database or DynamoDB) and writes config/cache.ts.

With the file store, that file imports env without using it, which tsc refuses: delete the import env from '#start/env' line.

Cache in a service

app/services/blog_service.ts

import { inject } from '@adonisjs/core'
import cache from '@adonisjs/cache/services/main'
import { SiteClient } from '@fetchski/attract-cms-sdk'

@inject()
export class BlogService {
  constructor(protected site: SiteClient) {}

  /** The first page of posts, read from the site at most every 10 minutes. */
  async latest() {
    return cache.getOrSet({
      key: 'attract:blog:latest',
      ttl: '10m',
      grace: '6h', // the site is down: keep serving the last answer
      factory: async () => {
        const list = await this.site.blogPosts().listBlogs({ perPage: 10 })
        return list.results.map((post) => ({ id: post.id, title: post.title, slug: post.slug }))
      },
    })
  }
}

The cache keeps plain JSON, so the factory returns the fields you show, not the model (a model’s Dates would come back as strings).

A readiness check

AdonisJS’s health checks answer /health/ready for your load balancer. Add one that asks the site a small read:

Set up health checks

Terminal

node ace configure health_checks

It writes start/health.ts and app/controllers/health_checks_controller.ts. Route them:

start/routes.ts

router.get('health/live', [controllers.HealthChecks, 'live'])
router.get('health/ready', [controllers.HealthChecks, 'ready'])

Its live() calls response.ok() without an argument, which tsc refuses: make it response.ok({ status: 'live' }).

Add the site's check

app/health_checks/attract_check.ts

import app from '@adonisjs/core/services/app'
import { BaseCheck, Result } from '@adonisjs/core/health'
import type { HealthCheckResult } from '@adonisjs/core/types/health'
import { SiteClient } from '@fetchski/attract-cms-sdk'

/** Ready when the site answers one small read the key may make. */
export class AttractCheck extends BaseCheck {
  name = 'Fetchski Attract site'

  async run(): Promise<HealthCheckResult> {
    const site = await app.container.make(SiteClient)
    const started = Date.now()
    try {
      await site.blogPosts().listBlogs({ perPage: 1 })
      return Result.ok(`${site.slug} answered`).mergeMetaData({ ms: Date.now() - started })
    } catch (error) {
      return Result.failed(`${site.slug} did not answer`, error as Error)
    }
  }
}

start/health.ts

import { HealthChecks, DiskSpaceCheck, MemoryHeapCheck } from '@adonisjs/core/health'
import { AttractCheck } from '../app/health_checks/attract_check.js'

export const healthChecks = new HealthChecks().register([
  new DiskSpaceCheck(),
  new MemoryHeapCheck(),
  new AttractCheck().cacheFor('1 minute'),
])

Pick a read your key’s scopes allow (this one needs content:blog:read). cacheFor keeps a load balancer’s frequent polls from spending the key’s rate limit.

The reference

Back to the overview, or open any method in the sidebar.

Get started

Using the SDK

Going further

Pages & Site

Content

Media

Forms

Ads & Conversions

Analytics

Queue jobs

Cached answers

A readiness check

Queue jobs

Cache the answers

A readiness check

The reference