Migrations¶

create_all() bootstraps a schema and never alters one — changed columns, removed fields, new indexes and changed types are silently ignored. viur-models therefore embeds Alembic:

pip install "spltz-viur-models[migrations]"

Project layout¶

The scaffold is generated — pass migrations= to viur.models.setup and the dev server writes whatever is missing on boot, then brings the database up to date:

# deploy/main.py, after core.setup()
viur.models.setup(migrations=models_db.PROJECT_ROOT)

This replaces a one-off alembic init: missing scaffold files are regenerated, existing ones never overwritten (viur.models.scaffold). The scaffold goes into the project root, next to the distribution folder:

myproject/
  alembic.ini
  migrations/
    env.py              # ~4 lines, see below
    script.py.mako
    versions/           # the revisions — commit these, they are code
  deploy/               # the distribution folder = what gets deployed
    models_db.py        # the DB settings — framework-free, read by BOTH sides
    models/             # the models
    main.py

Why outside deploy/

Revisions are applied by a deploy step or by hand, never by the running app — which only reports its schema state via viur.models.setup. A scaffold inside deploy/ works as well, with prepend_sys_path = %(here)s.

What gets generated¶

`alembic.ini`¶

Two entries matter; the rest can stay as it is.

[alembic]
script_location = %(here)s/migrations

# Puts the distribution folder on sys.path so env.py can import the
# project's ``models`` package and ``models_db``. With the scaffold inside
# deploy/ this is just %(here)s.
prepend_sys_path = %(here)s/deploy

# Left empty on purpose — resolve_url() below decides.
sqlalchemy.url =

`env.py`¶

The project file states which models to migrate and where the database is by default; everything else is viur.models.migrations:

from viur.models import migrations
import models_db

# Populates SQLModel.metadata. Without this, autogenerate sees an empty
# schema and generates a migration that DROPS EVERY TABLE.
migrations.import_models("models")

migrations.run(fallback_url=models_db.url())

env.py must not need viur-core

alembic runs from a shell without the App Engine stack — hence the settings in a plain module (models_db.py), not behind conf.models.

`script.py.mako`¶

Carries import sqlmodel: autogenerate renders SQLModel's column types by full path (sqlmodel.sql.sqltypes.AutoString(length=50)).

Where the database URL comes from¶

resolve_url, most explicit first:

#	Source	Use
1	`alembic -x url=…`	one-off override
2	`$VIUR_MODELS_DSN`	CI, deploy steps
3	the engine configured in this process	migrating from a booted app or a test fixture
4	the `conf.models` preset	after an app boot
5	`fallback_url` passed by `env.py`	the project default
6	`sqlalchemy.url` in `alembic.ini`	pinning one database

A misconfigured preset (postgres without a DSN) raises instead of falling through.

The first revision¶

A freshly generated scaffold gets one autogenerated revision, applied immediately:

Starting point	What happens
Empty database	The revision creates every table; `upgrade head` applies it.
Database that already has the tables (an earlier `create_all()`)	The revision is autogenerated against a throwaway empty database, so it still describes the full schema — and the real database is stamped, not upgraded. Nothing is dropped, nothing recreated.

Autogenerating against the populated database would yield an empty revision that could never rebuild the schema — hence the throwaway database. initial_revision=False leaves the first revision to you.

Daily use¶

Run from wherever alembic.ini lives — the project root in the layout above:

alembic revision --autogenerate -m "add slug to entry"
alembic upgrade head
alembic current
alembic downgrade -1
alembic check          # do models and schema still agree?

Always read the generated revision

A column rename is drop + add to Alembic — fix by hand to op.alter_column(..., new_column_name=…). (A relation rename in a multiple↔single switch is paired, see below.)

alembic check is the CI gate: it fails when a model changed without a matching revision.

Bone-level transitions are generated for you¶

In a skeleton project multiple=True → False is coerced on read (BaseBone.unserialize takes loadVal[0]); in SQL it is a link table becoming an FK column — data Alembic would drop first. viur-models detects these transitions and generates the data migration by viur-core's own rules:

Change	Generated	Rule, and where it comes from
new field	`fill_column` (only if required)	`getDefaultValue`, taken from the model's own default
field removed	plain `drop_column`	the loss is intended
new field in a link model (`using`)	`fill_column` on the link table	same as a new field
`str` ↔ `Text`	`coerce_text`	`StringBone.type_coerce_single_value`, never truncates
numeric precision	`coerce_numeric`	`NumericBone._convert_to_numeric`
multiple ↔ single	`collapse_multiple` / `expand_multiple`	`loadVal[0]` — "take the first one"; `expand_multiple` fills `NOT NULL` payload columns from the model defaults (`Ellipsis` stub if none)
multilingual ↔ plain	`reduce_languages` / `expand_languages`	see the asymmetry below
`select` → `bool`	`coerce_bool`	`parse.bool` with `conf.bone_boolean_str2true`
`bool` → `select`	`remap_values` stub	viur-core has no rule — see below

The generated revision already contains the data migration:

def upgrade() -> None:
    with op.batch_alter_table("post") as batch_op:
        batch_op.add_column(sa.Column("slug", sa.String(60), nullable=True))

    op.coerce_text("post", "body", new_type=sa.String(), nullable=False)
    op.fill_column("post", "slug", "", nullable=False)
    op.reduce_languages("post", "title", new_type=sa.String(200),
                        languages=["de", "en"], keep="de", nullable=True)
    op.collapse_multiple("post", link_table="post_tag", target_column="tag_id",
                         link_parent_fk="post_id", link_dest_fk="tag_id",
                         foreign_table="tag", target_type=sa.Integer(), keep="first")

Autogenerate explains each decision on stdout; the row count of a reduction is reported when the revision runs:

viur-models: post.title: multilingual -> single value (keeps de)
viur-models: post.tags -> tag: multiple -> single (keeps the first target)
viur-models: post.tag_id: 3 row(s) had several targets — kept the first one, like the bone does

The two language directions are asymmetric

Reducing keeps the field's first declared language (stand-in for conf.i18n.default_language, which a migration cannot read); expanding puts the value under languages[0]. That asymmetry is viur-core's (BaseBone.unserialize).

bool → select needs one line from you

SelectBone.singleValueUnserialize matches by value; a stored True matches nothing. The generator emits a stub with the stored labels and the revision refuses to run until it is filled:

op.remap_values("post", "flag", {True: ..., False: ...},
                new_type=sa.Enum("YES", "NO", name="kind"))

select → bool is automatic (parse.bool).

A rename across the switch (tags → tag) is paired by target table; only an ambiguous pairing (several relations onto the same target) falls back to add/remove.

How the detection works¶

Alembic sees DDL only: str → Text drops max_length, which its type comparison reads as "no opinion" (nothing reported); multiple → single is two unrelated operations. Detection therefore runs on structure snapshots (structure_for_model() per table model, written to migrations/structures/<revision>.json on every autogenerate) diffed against the parent revision's snapshot.

Commit the snapshots. A missing one yields an empty diff — adopting them later is safe. Baseline for an existing revision:

from viur.models import migrations, schema

migrations.import_models("models")
schema.save(schema.snapshot_dir("migrations"), "<head revision>", schema.snapshot())

Custom column types¶

render_item renders a TypeDecorator as its impl — RecordJSON becomes sa.JSON(): a revision is a frozen snapshot and must not import a model class. SQLModel's own decorators (AutoString & co.) are left to Alembic.

What runs when¶

The app never creates or migrates the schema. viur.models.setup reports the revision via schema_revision (plain SQL, no Alembic import) and runs create_all() for the memory preset only.

Never migrate at instance start

Several App Engine instances would migrate the same database at once. Migrations are a one-off step before the rollout.

Adopting an existing database¶

A database bootstrapped with create_all() has no alembic_version table. setup(migrations=…) adopts it on the next dev-server boot without DDL. By hand — a revision describing the schema must exist first, stamp only marks:

alembic stamp head
alembic check      # confirms it really matches

SQLite¶

Batch mode is on for SQLite URLs (with op.batch_alter_table(...) in revisions) — SQLite cannot ALTER most things.