NOC 19.3 is Released¶
ConfDB Object and Interface Validation Policies¶
NOC 19.3 brings ability to validate object and interface settings using ConfDB Queries. This feature will became primary way of validation and will replace old CLIPS-based implementation in NOC 19.5.
It is good time to begin legacy validation policy migrations.
ConfDB Syntax Expansion¶
ConfDB got additional syntax for:
- NTP support
- Video, Audio settings and media streaming
ConfDB 'raw' Section¶
Raw tokenized input can be applied to ConfDB under the
raw node. So platform-depended syntax can be processed via ConfDB Queries and validators natively, even if no platform-independentent syntax exists.
raw section may be enabled on Managed Object Profile level.
ConfDB 'meta' Section¶
NOC 19.3 adds
meta section, containing valuable NOC database information directly exposed to ConfDB queries. Lots of additional information included:
- Software Version
- Administrative Domain
- Network Segment
- Managed Object Tags
- Managed Object Profile
- Interface Profile
- Interface Network Neighbors (i.e. Links)
NOC 19.3 introduces 4 new profiles:
Uplink detection algorithm for RCA became configurable now. Following policies may be used:
- Segment Hierarchy (previous algorithm)
- Object Level
- All Segment Objects
- Lesser Management Address
- Greater Management Address
Uplink policies may be configured at Network Segment Profile level. Multiple policies may be used with falling back to next policy until uplinks are detected.
Uplink policies are greatly improve the quality of topology-based RCA. See Uplink Policy for details.
Topology RCA Optimization¶
Topology-based Root-Cause analysis may be resource consumption. NOC 19.3 introduces new experimental accelerated mode called
RCA Neighbor Cache. Smarter data precalculation and caching in combination of database query optimization and bulk updates allows to achieve 2-3 times speedup on real-world installations.
To enable the feature perform following steps:
- Run fix:
$ noc fix apply fix_object_uplinks
- Enable enable_rca_neighbor_cache configuration variable
Alarm processing will be postponed when correlator process is stopped, so alarm creation and clearing will be delayed until the correlator process will be started again.
Prometheus histograms and quantiles may be exported via /metrics endpoint. Additional metrics may be enabled in config. See metrics section for details.
Previous releases of NOC relied on venerable Django 1.4 dated back to 2012. Django's team worked hard on improving their product according to their vision. Unfortunately they tend to introduce a lot of incompatibilities and upgrading to each next Django's major release is the real pain. Django 1.4 fits our needs well but is not maintained and is incompatible with Python 3. So it is the time to to collect the pains.
We'd migrated from 1.4 to 1.5, then from 1.5 to 1.6, then followed by upgrades to 1.7, 1.8, 1.9, 1.10 and stopped at 1.11. During our stroll we'd became very disappointed by Django's API stability and the high maintenance costs for the complex applications and applied some countermeasures.
NOC 19.3 brings following changes:
- Django 1.11.22
- Django's auth contrib package has been replaced with
Southmigrations has been replaced with our own
- All legacy Django admin applications (ModelApplication) has been replaced with ExtJS implementations.
- Django will never create or modify database structure on its own (so-called syncdb).
- Django static media repacked as django-media npkg package.
User and Groups use NOC's own implementation instead of Django's ones. Besides the native ExtJS UI it allows future independentent development according our needs. User Profile became the part of User model.
South database migration engine stopped in development and users are urged to move to Django's 1.7 built-in migration engine. During our investigations we'd found that we need to completely rewrite 500+ of existing migrations, migrations code will be bloated by the unnecessary abstractions and we need to invite the way to preserve old migration history.
So we'd developed migration engine, simple but powerful. Key benefits are:
- Small, clean API.
- Semi-automatical translation of existing migration.
- Seamless migration history conversion.
- Skipped migrations with from other development branch, may be applied later.
Development Process Changes¶
NOC adopts black - the python code formatter. CI pipeline checks code formatting of changed python files. Any misformatting considered the error and CI pipeline fails at the [lint` stage. We recommend to add black formatting to git's pre-commit hook or to the IDE's on-save hook.
We'd already reformatted all ours codebase and NOC is now fully `PEP8](https://www.python.org/dev/peps/pep-0008/)-compatible. Docker container is also available. Use:
docker run --rm \ -w /src \ -v $PWD:/src \ registry.getnoc.com/infrastructure/black:master \ /usr/local/bin/black <file name>
to format file
Towards Python 3 compatibility¶
Python 3 compatibility became one of our priorities. With 19.3 we'd fixed lots of incompatibilities, upgraded same dependencies and becoming to get rid of unsupported ones. Though a lots of work and testing still required we're expecting to reach full Python 3 compatibility in one of future releases.
We're developed the policy for Merge Request's (MR) labels. CI pipeline checks the labels and fails at the
lint stage in case of errors. Label policy helps to organize testing and code reviewing process and quickly explains the goals of MR and subsystems affected.
Explicit MongoDB Connections¶
Prior to 19.3 NOC relied that importing of
noc.lib.nosql automatically creates MongoDB connection. This kind of auto-magic used to work but requires to access all mongo-related stuff via
noc.lib.nosql. Starting from 19.3 we're beginning to cleanup API and the code and demand, that MongoDB connection is to be initialized implicitly.
For custom commands and python scripts
from noc.core.mongo.connection import connect ... connect()
For custom services set service's
use_mongo property to
- ManagedObjectSelector.resolve_expression() renamed to ManagedObjectSelector.get_objects_from_expression()