adjust

All code for adjust.

What can users do now?

1. They can import their books.

• the books will have their title inferred if possible,
• the books will have a short description inferred,
• the books will be indexed in sqlite3 full text search,

2. Asynchronous pipelines transition:

• pending if have pending tasks,
• failed if 1 out of N tasks fail after X attempts,
• success if all succeed,
• input and output artifacts of every tasks are written,
• some tasks have side-effects (like indexing, editing title and description),

3. We can view /sqlite3 tables and sort them by all column values,
4. we can view /runs and /run and see DAG and status,
5. we can view /users
6. adjust_table allows sorting by columns and if sort param is exposed, it should be clickable,
7. login works with email and password,

• when session expires we keep the email to help the user login once again,
• session expires every X hours,
• no endpoints except / and /login are accessible without a session,

8. http server supports 1.1 and keep-alive,
9. validation.py library serves similarly to pydantic, but narrower scope,
10. pipeline library allows typesafe creation of pipelines, which are:

• decomposed into tasks,
• have stable ids so that we can easily write inputs and outputs,
• TODO somehow we currently create fake functions to submit pipelines, it would be nice if we can use actual task handlers, so if any of the task handlers change, pipeline submission throws a type error if some other task handler did not.

11. Pipeline concurrency invariants (important for future PostgreSQL support):

• never do read-modify-write counters in Python for task/group state,
• keep state transitions and counter updates in one SQL transaction,
• use guarded transitions (WHERE status IN (...)) so a task is terminalized once,
• use additive SQL updates (x = x + 1, x = x - 1) for counters,
• make close/finalize updates idempotent (... WHERE status='open' AND ...).

12. many tests spawn a server and do e2e testing for all state transitions:

• html snapshots are enough because css and JS are inside the single html page and can be opened without a server to check if it works,
• html snapshots track all state transitions as all of them should be visible as backend systems progress.

Users

User types

Regular

Can submit search queries for a given database of documents.

Admin

Can access HTTP endpoint in the account context. Can manage users related to its context.

Superadmin

Can access all HTTP endpoints in all contexts.

HTTP server

All methods require authentication. If there is no valid session, redirect to /login is fine.

Local server

Use config-driven compose:

• make docker-up ENV=local TARGET=cpu
• make docker-up ENV=local TARGET=gpu

Ollama (local LLM)

This project can talk to a local Ollama server over HTTP without extra Python dependencies.

Quick start:

• Install Ollama and make sure the ollama CLI is on your PATH.
• Start the Ollama server (desktop app or ollama serve).
• Pull configured model: make ollama-pull
• Smoke test from config/scripts.toml: make ollama-smoke

Change smoke endpoints/prompts in config/scripts.toml.

Docker (CPU + GPU)

Primary path uses one TOML config plus generated env:

• Source config: config/config.toml
• Compose file: docker-compose.config.yml
• Generated env: build/config.<env>.<target>.env
• Generated vLLM config: build/vllm.<env>.<target>.yaml

config/config.toml is the single source of truth. make config-compile turns it into concrete runtime files for a selected ENV and TARGET.

Bring stack up/down from config:

• make docker-up ENV=local TARGET=cpu
• make docker-up ENV=local TARGET=gpu
• make docker-up ENV=prod TARGET=cpu
• make docker-up ENV=prod TARGET=gpu
• make docker-down ENV=local TARGET=gpu

Print compiled env for inspection:

• make config-print ENV=local TARGET=gpu

Download a vLLM model with temporary internet egress:

• make vllm-download ENV=local TARGET=gpu
• stop it when done, then make vllm-download-down ENV=local TARGET=gpu

Notes:

• CPU/GPU choice is controlled by TARGET.
• Local/prod differences are controlled by ENV.
• COMPOSE_PROFILES is generated from TOML toggles, including service enable/disable.
• docker-up defaults: ENV=local runs attached (--build), non-local (for example ENV=prod) runs detached (-d --build).
• Runtime vllm is offline by default (HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1) and on the internal network.
• vllm-download adds docker-compose.vllm-egress.yml, flips offline flags to 0, and enables temporary egress for model pulls.
• vllm_cpu and vllm_gpu both use ${VLLM_IMAGE} and mount the same named volume vllm:/root/.cache/huggingface. Downloading models once makes them available to both CPU and GPU profiles on the same host.

Smoke Scripts

Smoke script config lives in config/scripts.toml.

• make ollama-smoke SCRIPTS_PROFILE=local
• make vllm-smoke SCRIPTS_PROFILE=local
• make vllm-tokenize-smoke SCRIPTS_PROFILE=local

Logging

By default logs go to stdout/stderr. You can also write to a rotating file:

• ADJUST_LOG_PATH=/var/log/adjust.log
• ADJUST_LOG_MAX_BYTES=10000000 (default 10MB)
• ADJUST_LOG_BACKUPS=5 (default 5)

Skills for agents

Sometimes they are also good instructions for humans on how to contribute.

Secrets

/etc/adjust/secrets/ with docker references for now. in the future, ansible with systemd env variables

For vLLM Hugging Face downloads, place a token at /etc/adjust/secrets/hf_token on the host. The compose files will read it if present.