Delegation checker and automatic limit_domains management - #1235
Delegation checker and automatic limit_domains management#1235peterthomassen wants to merge 15 commits into
Conversation
adabf3d to
245e4e0
Compare
peterthomassen
changed the title
|
Extended PR with automatic domain limit management. Effective domain limit is number of domains securely delegated to us ( For users without .dedyn.io domains, this is equivalent to (e + sqrt(e)); .dedyn.io domains are added on top (effectively not influencing the limit besides offsetting it). Illustrations: Account without dedyn.io domainAll secure domains
Account with dedyn.io domainSecure domains
|
6560072 to
889ad0e
Compare
|
I've performed a detailed/thorough review, and applied various improvements / rearranged commits. I did not find any problematic issues. The first two commits initially might seem unrelated, but they are needed so that the later limit calculation logic works reliably. Perhaps we should drop the commits containing the plans. @nils-wisiol ready for final review |
PR 1235 Review: Delegation Checks and Automatic Domain LimitsCompared: Checklist
|
|
DomainSerializer rejects a name below one of our local public suffixes that is not an immediate child of it, such as sub.example.dedyn.io, with the name_too_deep code. Signup reports that message as it stands instead of replacing it with the generic name_unavailable one.
Creating a second domain under one of our own public suffixes is now rejected, unless support has pinned the account's domain limit; accounts that already hold several keep them. DomainQuerySet gains under_local_public_suffix(), built on a new under_local_public_suffix_q() that matches at any depth below one of our own public suffixes.
Describes how a hosted zone is checked for being correctly delegated to deSEC and correctly secured, and how the outcomes are recorded: what is measured and from where, where the code lives, what is stored, and what is deliberately deferred.
New compose service running a validating recursive resolver, built from source for its flush_delegation command. It sits alone with api on the new rearapi_unbound network, serving DNS on port 53 and remote control on 8953.
desecapi.unbound speaks unbound's remote control protocol: flush_delegation() drops a name's delegation from the resolver's cache, and query() sends it a recursive, DNSSEC-enabled query. Settings gain UNBOUND_HOST, UNBOUND_PORT and UNBOUND_CONTROL_PORT.
New DelegationCheck model holding the outcome of one check: the two status dimensions, the nameservers seen, and the rcode and EDE of the response that settled the security status. DelegationCheck.objects.record() bumps the confirmation time when an outcome agrees with the domain's current one, and inserts a new row otherwise. Domain gains current_delegation_check, pointing at the newest one.
desecapi.delegation.check() returns the nameservers a name is delegated to and whether that delegation is secured, as a DelegationCheckResult that touches no database. It flushes the delegation from the resolver's cache, locates the parent zone, reads the delegation from the parent's nameservers, and takes the security status from our resolver's AD bit. desecapi.dns gains query_server() for the non-recursive queries to authoritative servers.
Checks the named domains, all of them (--all), or those not checked within a given number of seconds (--stale), records the results, and prints one line per domain. --include-local extends bulk selection to locally registrable domains, which are otherwise skipped; --dry-run prints without recording; --concurrency sizes the thread pool.
Describes how the domain limit becomes a function of the domains a user has securely delegated to us, how checks are scheduled and triggered, which decisions were taken along the way, and what is deferred.
A null User.limit_domains -- now the default -- makes the enforced limit s + max(DOMAIN_LIMIT_INSECURE_HEADROOM, round(sqrt(e))), with s the user's securely delegated domains and e the externally delegated ones; an explicit value still pins it. LIMIT_USER_DOMAIN_COUNT_DEFAULT is renamed to DOMAIN_LIMIT_INSECURE_HEADROOM, its environment variable unchanged, and now floors the headroom rather than the limit. Domain.secure_delegation_since, maintained by DelegationCheck.objects.record(), is what those counts read. The account endpoint keeps reporting limit_domains as the enforced number and gains secure_domains. Migration 0048 adds the field, backfills it from the checks already recorded, and makes limit_domains nullable by default.
Support can pin an account's domain limit by setting a number; there was no way back. "auto" writes null, which is what the computed limit is spelled as, and the command reports the value that results rather than the null it wrote.
New celery-delegation service, a second instance of the api image consuming the delegation_bulk and delegation_adhoc queues at one domain per task. desecapi.tasks holds check_domain_delegation, which re-checks staleness before measuring and leaves the domain for later when the resolver is unavailable, and plan_user_delegation_checks, which puts a user's not-yet-secure domains on the ad-hoc queue, capped at 20 per plan and one plan per user per minute.
check-delegation with --concurrency 0 enqueues one task per selected domain on the bulk queue instead of checking them inline, reusing the same selection and passing --stale on so the workers can re-check it; --dry-run reports what would be enqueued. Negative values are rejected.
A cron entry in the api container runs check-delegation --stale 79200 --concurrency 0 every two hours, which gets each domain re-checked about once a day.
Creating a domain, and being denied one by the domain limit, both enqueue checks for the user's domains that are not yet known to be secure. Bounded to one plan per user per minute and 20 domains per plan; neither trigger raises. The limit message now says that a re-check is under way.
889ad0e to
0c9716f
Compare
It does not take precedence, but workers consume both queues round-robin, and bulk tasks are enqueued per domain. Thus, when an ad-hoc check task is added, a worker will pick it up quickly, then go back to pending bulk tasks. Effectively, adhoc queue gets half the slots as long as it is not empty. Nothing to be gained by separating workers. Hard cut-off for a check is 30s, expected time for any slot to free up is < 2s.
Increased from 4 to 8. Potential constraints are TLD nameserver rate limits (probably far away), queuing within unbound (we give it
Done. |
2 participants
Different approach than #1174
Testing using production database on local machine takes about 79 minutes with 4 threads, 47 minutes with 8 threads, 44 minutes with 16 threads, and 39 minute with 64 threads. 8 threads is probably best: the more threads, the more resolution errors (maybe rate limits at TLD auths or something).