Operations

Running a Zeyn app in production: time zone data, signed licenses, shared state for several servers, logs and traces, PDF documents and e-mail.

Time zones

TimeZone::named accepts every IANA zone (Europe/Istanbul, America/New_York, Australia/Sydney, in any case) and fixed offsets (UTC+03:00, +05:30).

• Zone data is the IANA time zone database built into the binary through jiff (about 200 KB). Answers are the same on every platform, Windows included, and no system zoneinfo is read.
• History is modeled: Turkey's daylight saving until 2016 and its permanent +03 since, the United States rules before 2007, and every other recorded change.
• A wall clock time that does not exist (clocks moved forward) is refused with LocalTimeError::Skipped. One that happens twice (clocks moved back) means the earlier moment.
• A zone is read from the bundle the first time its name is used and kept for the life of the process, a few kilobytes each. Unknown names are never kept, so clients cannot grow the cache.
• TimeZone::names() is a short list of common zones for the shell's zone picker. A user's zone that is not on the list is added to the picker for that user.
• New zone rules arrive with a newer jiff release (cargo update -p jiff-tzdb). Rebuild and deploy when a country changes its rules.

Signed licenses

A license file is signed with an ed25519 key that only the vendor has. The app embeds the public key, so a customer can read a license but cannot change it or make a new one. Turn it on with the signed-licenses Cargo feature.

zeyn license keygen                      # writes zeyn-license.key, prints the public key
zeyn license sign acme.json --out license.json

acme.json holds the license fields:

{ "holder": "Acme", "valid_until": "2027-12-31", "modules": ["pos", "stock"], "max_users": 50 }

const LICENSE_KEY: &str = "f414a89e9dae57598d60141df96dd1c9fb655fdcf49f5eaf24e093fda7d332e3";

App::new("Shop")
    .license(SignedLicenseFile::new("license.json", LICENSE_KEY), Duration::from_secs(3600))
    .feature(Feature::new("pos").guard(Licensed::module("pos")).screen(pos))

• The file is JSON with the license as text in payload and the signature in signature. What is signed is zeyn-license-v1\n followed by the payload text, so the signed bytes are exactly the stored bytes.
• The private key stays on the signing machine. keygen never overwrites a key file, and *.key is in .gitignore of new projects. sign reads the key from --key, from the ZEYN_LICENSE_KEY variable or from zeyn-license.key, and refuses a license without holder or with a valid_until that is not YYYY-MM-DD.
• A changed file, a file signed with another key or a file that is not a license is refused: License::from_signed returns LicenseError (BadSignature, InvalidPublicKey, Malformed), the server logs why, and every Licensed guard denies with "Your license does not cover this." A file that cannot be read keeps the last license, like any failing license source.
• Expiry is checked by the guard on every request against the user's today, so an expired license locks the app with "Your license has expired." without a restart. A renewed file is read at the next refresh.
• ed25519 cannot tell a changed file from one signed with another key; both are a signature that does not match, and the message says so.
• max_users is carried and signed but not enforced by the framework; the app decides what it counts as a user.

Several servers: shared state in Redis

With the redis Cargo feature, RedisStore keeps the state that must be the same on every instance behind a load balancer:

let redis = RedisStore::new(config.require("redis.url")?, "billing").map_err(Error::Internal)?;
App::new("Billing")
    .rate_limit(RateLimit::per_ip(300, Duration::from_secs(60)))
    .rate_store(redis.clone())
    .datagrid_views(redis)

What the server keeps per user, and where:

• Rate limits are a token bucket in one Lua script, so concurrent requests on different servers never exceed the limit. Time comes from Redis, not from the app servers. Counters expire one window after their last use.
• Views are JSON under zeyn:<prefix>:view:<user length>:<user>:<grid> and never expire.
• The connection is made on first use and comes back by itself. Calls time out after 500 ms. When Redis is down, rate limits let requests through, views are neither loaded nor saved (the grid falls back to its own layout, the dock to localStorage), the outage and the recovery are each logged once, and Redis is left alone for five seconds after each failure so requests do not each wait for a timeout.
• RateStore::take returns a future so stores can talk to a network. MemoryRateStore::take stays synchronous for direct use.
• TLS (rediss://) is not built in.
• Tests run against a real server when ZEYN_TEST_REDIS is set:

  docker run -d --rm --name zeyn-redis -p 6379:6379 redis:7-alpine
  ZEYN_TEST_REDIS=redis://127.0.0.1:6379 cargo test -p zeyn --features redis --test redis
  docker stop zeyn-redis

Logs and traces

The framework reports through tracing. It never prints by itself.

• App::serve and App::serve_until install a subscriber unless the app installed one first: events go to standard error as text lines, filtered by log.level (info by default; any tracing target filter such as warn,zeyn=debug). An app that wants JSON lines or another sink installs its own subscriber before serving, and Zeyn leaves it alone.
• At info the output is what it always was: the listening address, startup warnings (missing translations, DevAuth, dev_latency), failing license sources and internal errors. At debug every call adds a call finished line with kind, screen, action, user id, tenant id, outcome, error code and milliseconds.
• Spans, for tracing backends:

• With the otel Cargo feature, spans are exported over OTLP/HTTP when otel.endpoint is set in Config (http://collector:4318; spans go to /v1/traces). otel.service_name names the service, the app's name by default. Spans are batched on a background thread and flushed when serve_until stops.
• Without export, standard error receives events only, so no span is built and the spans cost one cached check each. Apps mounted with App::router in another server, and tests, pay the same.
• What is never recorded: field values (masked or not), form and API input, query strings, cookies, headers and secrets. A test runs grids, form saves, API calls and reveals under a trace level subscriber and checks that no masked value appears. Two things stay with the app: paths are recorded, so do not put personal data in routes, and Error::Internal messages are logged as written.

PDF documents

The pdf Cargo feature renders PDF documents from Typst templates. It is off by default because Typst is large (see the numbers below).

const INVOICE: PdfTemplate = PdfTemplate::new("invoice", include_str!("../templates/invoice.typ"));

#[action(permission = "sales.order.read")]
pub async fn invoice_data(p: OrderParams, store: Orders, ctx: Ctx) -> Result<InvoiceData> { ... }

Screen::window("Order").body(PdfLink::new("Print invoice", INVOICE, invoice_data, params))

App::new("Shop").service(Pdf::from_config(&config)?)

• PdfLink puts a download button on a screen. GET /_zeyn/pdf opens the screen with its access checks, runs the action for the user (its permission and interceptors apply, the call kind is Document) and renders the template with what the action returns. Users who may not run the action do not see the button.
• Grids with exportable() also print a list report from the toolbar's Export::Pdf (GET /_zeyn/grid/export?format=pdf): the user's columns, labels, filters and sort, numbers aligned right, a repeated header row and page numbers. A report holds at most 2,000 rows (MAX_PDF_ROWS, or the grid's export limit when smaller) and says so when rows were left out.
• Data never becomes Typst code. The action's output is serialized to JSON and the template reads it as data (the renderer binds #let data = json("/data.json") before the template). Values such as #panic(), ] or $x$ are printed as text; a test checks this. Templates can read nothing else: no files, no packages.
• Rendering runs on a blocking thread. At most pdf.max_concurrent documents (1 to 64; two by default, one on a single CPU) render at the same time; the rest wait for a slot. Typst's memoization cache is emptied after each document, so memory goes back down.

Fonts: Libertinus Serif (regular, bold, italic, bold italic; SIL Open Font License) is embedded and covers Turkish; DejaVu Sans Mono is embedded as the fallback for signs Libertinus lacks, such as ₺. Their licenses are in crates/zeyn/assets/fonts. There is no math font; add one with pdf.fonts if templates set equations.

Measured on Windows (release build, x86_64):

A debug build is about ten times slower (a 2,000 row report took 71 s in the showcase).

E-mail

The email Cargo feature sends notifications with lettre.

let mailer = Mailer::from_config(&config)?;
App::new("People").mailer(mailer)

#[action(permission = "people.leave.request", audit)]
pub async fn move_leave(r: TransitionRequest<LeaveRequestId>, store: Leaves, mailer: Mailer, ctx: Ctx) -> Result<LeaveState> {
    let state = leave_flow().apply(&ctx, current, &r.transition, r.comment.as_deref())?;
    mailer.send(&ctx, Mail::transition(&ctx, &manager_email, &employee)?)?;
    Ok(state)
}

• Mailer::send renders the message and puts it in an in memory queue (at most 10,000 messages), then returns. The zeyn.mail background job sends the queue at once, tries failed messages again with backoff (five times from ten seconds unless Mailer::retry says otherwise) and looks at the queue every minute. A message is dropped, with an error in the log, after eight failed tries. Like every job, a queue that has not been sent when the process stops is lost; durable delivery needs the database backed queue of the data layer.
• Messages come from the i18n catalogs: <template>.subject, <template>.text and optionally <template>.html, with {name} arguments. The recipient's language is the sender's unless Mail::locale sets it. Every message has a plain text and an HTML part; in the HTML part every argument is escaped, and without an .html entry the HTML is the escaped text. Line breaks are removed from subjects, and addresses are parsed, so data cannot add headers or recipients.
• Mail::transition writes the move the call just made with Workflow::apply (record, from, to, who and the comment) with the built in zeyn.mail.transition template. The showcase tells the manager when a leave request moves.

TLS uses rustls with the Mozilla root certificates (webpki-roots).