Reconciliation

How the bouncer keeps MikroTik address-list membership synchronized with CrowdSec decisions.

The reconciliation problem

When the bouncer starts (or restarts), or during the periodic reconciliation interval, the MikroTik address list may be out of sync with CrowdSec’s active decisions:

• IPs may have been added to CrowdSec while the bouncer was offline
• IPs may have expired in CrowdSec but still be in the MikroTik list
• IPs may have expired or been removed manually on MikroTik while still active in CrowdSec
• Previous bouncer entries may still exist from a prior run

By default, reconciliation runs at startup and then every 15m. Configure crowdsec.reconciliation_interval to change this cadence. Set it to 0 to disable periodic reconciliation; values below 1m are rejected at startup, so use 0 to disable or a duration of 1m or greater to enable.

Reconciliation process

1. Collect

   Fetch all active decisions from CrowdSec and all address-list entries from MikroTik.

2. Diff

   Compare the two sets to determine which IPs need to be added and which removed.

3. Refresh the cache

   Replace this family’s in-memory address cache with what the router just reported — before a single add or remove goes out, so the live ban and unban handlers stop trusting entries the router no longer has.

4. Apply

   Execute additions (via bulk scripts, 100 IPs/batch) and deletions (parallel API calls through the connection pool when it opened, one by one on the main connection when it did not). Each of the two patches the cache with what it attempted to write.

5. Verify

   Record the reconciliation metrics and log final counts to confirm address-list membership matches CrowdSec.

The diff itself is per protocol family and per address list. Two sets go in — what CrowdSec says should be blocked, and what the router currently holds — and two lists come out.

The prefix filter on the present set is what makes reconciliation safe to point at a shared list. The bouncer asks RouterOS for the list by name and then keeps only the entries whose comment starts with the configured comment_prefix, so an address you added by hand — or one another tool maintains — is not in the comparison at all, and the “present, not wanted” branch can never reach it.

The comparison is by normalised address, not by decision: several decisions for the same IP collapse to one wanted entry, and only the last one read supplies the timeout and comment written with it.

Performance

Measured on a MikroTik RB5009UG+S+ (ARM64, 4 cores @ 1400 MHz, 1 GB RAM, RouterOS 7.22.1) with mikrotik.pool_size: 10:

Metric	Full CAPI (~28,700 IPs)
Cold reconciliation wall-clock	~58 s
RouterOS bulk add work	~35–36 s
Router CPU peak observed	~39% during large add/remove churn
No-drift periodic check	~3–4 s, with no add/remove writes
CAPI → local-only mass removal	~26,800 removals in ~77 s of RouterOS removal work

Steady-state cost on current code

The no-drift row above belongs to the build and the list it was measured on. Two changes since then moved it: the reconcile pass stopped fetching two address-list properties nothing reads (−11.2%), and an individual unban stopped traversing the list to rediscover an id it already had. Re-measured on the same hardware running RouterOS 7.24.1, holding 22,857 entries (22,332 IPv4 plus ~525 IPv6) on a 60 s interval, taken from the bouncer’s own operation_duration_seconds histogram rather than from a stopwatch:

Metric	22,857 entries, RouterOS 7.24.1
No-drift periodic check	1.86 s over 28 consecutive cycles
Duty cycle at a 60 s interval	3.1% of wall-clock
Share spent in the two list reads	98.5%

What this means for tuning: the cost is dominated by reading the list, so it scales with how many entries the router holds, not with how much drift there is to repair. A pass that finds nothing to fix costs very nearly what a pass that fixes something costs. There is also a floor that no property reduction reaches — a count-only print of the same list still costs 1.18 s — so asking the router “has anything changed?” is not meaningfully cheaper than asking it for everything.

Router CPU

Router CPU spikes are expected during reconciliation because the bouncer is adding or removing many address-list entries. After reconciliation, RouterOS CPU should settle back near normal idle/traffic levels. Sustained high CPU usually indicates repeated API writes/reconnects or unrelated router workload, not the address-list entries simply existing in memory.

Periodic reconciliation is usually light: when there is no drift, it lists current address-list entries, compares them with the active CrowdSec snapshot, updates counters, and performs no add/remove writes.

Optimizations

Script-based bulk add

Instead of individual API calls (~97× slower), the bouncer uploads a temporary RouterOS script named crowdsec-bulk-import and runs it by its internal id:

/system/script/run =number=<script-id>

Each script run adds up to 100 IPs using :do { ... } on-error={} to handle duplicates gracefully.

Parallel deletion

Deletions use the configured connection pool with concurrent API calls. In the RB5009 CAPI test, removing ~26,800 CAPI-only entries took ~77 s of RouterOS removal work.

Pre-filtering

During initial decision collection, if a ban and corresponding unban are received for the same IP, the unban pre-filters the ban out of the pending set. This avoids adding and immediately removing the same IP.

Tip

If reconciliation takes too long, consider using origins filtering to limit the number of synced decisions. Local-only deployments are much smaller than full CAPI blocklists and cause less startup and restart churn.

Prometheus MetricsReconciliation-related metrics for monitoring.

Grafana DashboardVisual reconciliation monitoring.