Skip to content

Background Workers

Lucky Funatic runs several background workers that handle periodic tasks outside the request-response cycle. Workers can run alongside the API server (when ENABLE_BACKGROUND_WORKERS is set) or as a dedicated process using the -worker flag.

All workers implement a common interface with Start() and Stop() methods, and shut down gracefully on SIGTERM/SIGINT.

Worker Overview

flowchart LR
    subgraph Workers["Background Workers"]
        QCW["Quest Completion\n(every 1s)"]
        LPW["Leaderboard Persistence\n(every 5m)"]
        STW["Session Tracking\n(every 60s)"]
        PW["Popup\n(every ~3h)"]
    end

    QCW -->|"read schedule"| Redis["Redis"]
    QCW -->|"update quest status"| MySQL["MySQL"]
    QCW -->|"log state change"| Scylla["ScyllaDB"]

    LPW -->|"read leaderboard"| Redis
    LPW -->|"read entry counts"| Scylla
    LPW -->|"persist results & prizes"| MySQL

    STW -->|"read active sessions"| Redis
    STW -->|"save session history"| Scylla

    PW -->|"set popup flag"| Redis

Quest Completion Worker

Interval: Every 1 second Purpose: Processes auto-completing quests (wallet connections, exchange selection, etc.)

Some quests complete automatically when a server-side condition is met (e.g., the player connects a TON wallet). These completions are scheduled by adding entries to a Redis sorted set (quest_completion_schedule) with a score representing the scheduled completion time.

The worker polls this sorted set every second, picks up entries whose time has arrived, verifies the player actually meets the quest's completion criteria in MySQL, updates the quest status to "Completed", logs the state change to ScyllaDB, and removes the entry from the schedule.

This runs every second because quest completions should feel instant to the player -- they connect a wallet and the quest should show as completed within moments.

Leaderboard Persistence Worker

Interval: Every 5 minutes Purpose: Persists completed tournament leaderboards from Redis to MySQL and distributes prizes

While a tournament is running, its leaderboard lives in a Redis sorted set for real-time ranking performance. Once a tournament ends, this worker takes over:

  1. Finds tournaments that ended 15+ minutes ago and haven't had prizes distributed yet
  2. Acquires a distributed mutex (Redsync) to prevent duplicate processing across instances
  3. Reads the final leaderboard from Redis
  4. Queries ScyllaDB for entry counts per player
  5. Calculates prize distribution based on the tournament's rules
  6. Writes results and prizes to MySQL
  7. Updates winners' inventories with awarded items (TICO, Private Passes)
  8. Marks the tournament as prizes_distributed = TRUE

The 15-minute delay after tournament end gives late-finishing play sessions time to submit their final scores before the leaderboard is frozen.

Session Tracking Worker

Interval: Every 60 seconds Purpose: Persists active session data from Redis to ScyllaDB and cleans up expired sessions

Player sessions are tracked in-flight via the TrackSession middleware on every game route request. The middleware writes session start times to Redis. This worker periodically:

  1. Reads all active sessions from Redis
  2. Identifies expired sessions (no activity within the timeout window)
  3. Saves session records to ScyllaDB for analytics
  4. Removes expired sessions from Redis

Uses a distributed lock to prevent race conditions when multiple API instances run this worker.

Interval: Every 2 hours 40 minutes to 3 hours 20 minutes (randomized) Purpose: Controls the global popup display schedule

Manages a periodic popup system by:

  1. Setting a Redis key popup:active with a 5-minute TTL
  2. Resetting the view counter (popup:views)
  3. Scheduling the next trigger with a random offset (up to 20 minutes) to avoid predictable patterns

The randomized interval means popups appear roughly every 3 hours but players can't predict exactly when.

Worker Deployment

Workers can be deployed in two ways:

  • Embedded: Set ENABLE_BACKGROUND_WORKERS=true and workers run in goroutines alongside the API server. Simple but couples worker load with API load.
  • Dedicated: Run the binary with the -worker flag. This starts only the workers without the HTTP server. Allows independent scaling and isolation of worker processes.

Both modes support graceful shutdown -- on receiving a termination signal, workers finish their current cycle and clean up resources (database connections, Redis clients) before exiting.