Skip to content

NOC 19.3

In accordance to our Release Policy <releases-policy> we're proudly present release 19.3.

19.3 release contains 318 bugfixes, optimisations and improvements.

Highlights

New Profiles

  • Extreme.Summit200<profile-Extreme.Summit200>
  • Polygon.IOS<profile-Polygon.IOS>
  • Eltex.WOP<profile-Eltex.WOP>

ConfDB Object and Interface Validation Policies

NOC 19.3 brings ability to validate object and interface settings using ConfDB Queries<dev-confdb-query>. This feature will became primary way of validation and will replace old CLIPS-based implementation in NOC 19.5.

It is good time to begin legacy validation policy migrations.

ConfDB Syntax Expansion

ConfDB got additional syntax for:

  • NTP support
  • Video, Audio settings and media streaming

ConfDB 'raw' Section

Raw tokenized input can be applied to ConfDB under the raw node. So platform-depended syntax can be processed via ConfDB Queries and validators natively, even if no platform-independentent syntax exists. raw section may be enabled on Managed Object Profile level.

ConfDB 'meta' Section

NOC 19.3 adds meta section, containing valuable NOC database information directly exposed to ConfDB queries. Lots of additional information included:

  • Vendor
  • Platform
  • Software Version
  • Profile
  • Administrative Domain
  • Network Segment
  • Managed Object Tags
  • Managed Object Profile
  • Interface Profile
  • Interface Network Neighbors (i.e. Links)

ConfDB Normalizers

NOC 19.3 introduces 4 new profiles:

  • Beward.BD<profile-Beward.BD>
  • Cisco.IOS<profile-Cisco.IOS>
  • Dahua.DH<profile-Dahua.DH>
  • Hikvision.DSKV8<profile-Hikvision.DSKV8>

Uplink detection algorithm for RCA became configurable now. Following policies may be used:

  • Segment Hierarchy (previous algorithm)
  • Object Level
  • All Segment Objects
  • Lesser Management Address
  • Greater Management Address

Uplink policies may be configured at Network Segment Profile level. Multiple policies may be used with falling back to next policy until uplinks are detected.

Uplink policies are greatly improve the quality of topology-based RCA. See Uplink Policy<reference-network-segment-profile-uplink-policy> for details.

Topology RCA Optimization

Topology-based Root-Cause analysis may be resource consumption. NOC 19.3 introduces new experimental accelerated mode called RCA Neighbor Cache. Smarter data precalculation and caching in combination of database query optimization and bulk updates allows to achieve 2-3 times speedup on real-world installations.

To enable the feature perform following steps:

  • Run fix:

    $ noc fix apply fix_object_uplinks

  • Stop services-correlator processes

  • Enable config-fm-enable_rca_neighbor_cache configuration variable

  • Start services-correlator processes

Warning
Alarm processing will be postponed when correlator process is stopped, so alarm creation and clearing will be delayed until the correlator process will be started again.

Prometheus Histograms

Prometheus histograms and quantiles may be exported via /metrics endpoint. Additional metrics may be enabled in config. See metrics section<config-metrics> for details.

ObjectModel Tags

Inventory models got additional marking, which may be useful in various cases. See ObjectModel Tags<dev-objectmodel-tags> for details. Model's tags are also exposed into managedobject DataStream<api-datastream-managedobject>.

Django upgrade

Previous releases of NOC relied on venerable Django 1.4 dated back to

  1. Django's team worked hard on improving their product according to their vision. Unfortunately they tend to introduce a lot of incompatibilities and upgrading to each next Django's major release is the real pain. Django 1.4 fits our needs well but is not maintained and is incompatible with Python 3. So it is the time to to collect the pains.

We'd migrated from 1.4 to 1.5, then from 1.5 to 1.6, then followed by upgrades to 1.7, 1.8, 1.9, 1.10 and stopped at 1.11. During our stroll we'd became very disappointed by Django's API stability and the high maintenance costs for the complex applications and applied some countermeasures.

NOC 19.3 brings following changes:

  • Django 1.11.22
  • Django's auth contrib package has been replaced with AAA module<release-19.3-aaa>.
  • South migrations has been replaced with our own Migration Engine<release-19.3-migrations>.
  • All legacy Django admin applications (ModelApplication) has been replaced with ExtJS implementations.
  • Django will never create or modify database structure on its own (so-called syncdb).
  • Django static media repacked as django-media&lt;https://code.getnoc.com/npkg/django-media&gt; npkg package.

AAA module

User and Groups use NOC's own implementation instead of Django's ones. Besides the native ExtJS UI it allows future independentent development according our needs. User Profile became the part of User model.

Migration Engine

South database migration engine stopped in development and users are urged to move to Django's 1.7 built-in migration engine. During our investigations we'd found that we need to completely rewrite 500+ of existing migrations, migrations code will be bloated by the unnecessary abstractions and we need to invite the way to preserve old migration history.

So we'd developed migration engine, simple but powerful. Key benefits are:

  • Small, clean API.
  • Semi-automatic translation of existing migration.
  • Seamless migration history conversion.
  • Skipped migrations with from other development branch, may be applied later.

Development Process Changes

Code Formatting

NOC adopts black -the python code formatter. CI pipeline checks code formatting of changed python files. Any misformatting considered the error and CI pipeline fails at the lint stage. We recommend to add black formatting to git's pre-commit hook or to the IDE's on-save hook.

We'd already reformatted all ours codebase and NOC is now fully PEP8-compatible. Docker container is also available. Use:

docker run --rm \
    -w /src \
    -v $PWD:/src \
    registry.getnoc.com/infrastructure/black:master \
    /usr/local/bin/black <file name>

to format file

Towards Python 3 compatibility

Python 3 compatibility became one of our priorities. With 19.3 we'd fixed lots of incompatibilities, upgraded same dependencies and becoming to get rid of unsupported ones. Though a lots of work and testing still required we're expecting to reach full Python 3 compatibility in one of future releases.

MR Labels

We're developed the policy<dev-mr-labels> for Merge Request's (MR) labels. CI pipeline checks the labels and fails at the lint stage in case of errors. Label policy helps to organize testing and code reviewing process and quickly explains the goals of MR and subsystems affected.

Breaking Changes

Explicit MongoDB Connections

Prior to 19.3 NOC relied that importing of noc.lib.nosql automatically creates MongoDB connection. This kind of auto-magic used to work but requires to access all mongo-related stuff via noc.lib.nosql. Starting from 19.3 we're beginning to cleanup API and the code and demand, that MongoDB connection is to be initialized implicitly.

For custom commands and python scripts

from noc.core.mongo.connection import connect

...
connect()

For custom services set service's use_mongo property to True

Other Changes

  • ManagedObjectSelector.resolve_expression() renamed to ManagedObjectSelector.get_objects_from_expression()

New features

MRTitle
MR1682ClickHouse LowCardinality support
MR2091New migrations framework
MR2098Migration loader, planner and tests
MR2179Prometheus histograms
MR2181ObjectModels tags field
MR2190RCA neighbor cache and accelerated topology correlation
MR2220Merge UserProfile into User model
MR2228Platform tags
MR2245./noc test --idea-bookmarks option
MR2285Uplink Policy
MR2372Add MySQL Extractor
MR2400ConfDB: ntp support
MR2418#1077 ConfDB raw policy
MR2419Add new Profile Eltex.WOPLR
MR2420ConfDB: media section
MR2426ConfDB: Object Validation
MR2433ConfDB: meta section
MR2438ConfDB: Interface validation

Improvements

MRTitle
MR1888Django 1.5
MR1965Add input_vlan_map and output_vlan_map fields to ConfDB syntax.
MR2006noc/noc#1032
MR2020Add autowidth column option to ReportLinkDetail.
MR2021Add autowidth column to ReportIfacesStatus.
MR2023Add autowidth column option to ReportAlarmDetail.
MR2039Add frozen first row in Detail Report.
MR2041Add subscribers profile filter to AlarmDetail Report.
MR2048Add ability to get vlans from bridge on MikroTik.RouterOS
MR2055Add ignoring snmp to profile checker.
MR2065ensure-indexes: Create index on fm.Uptime
MR2078add get_mac
MR2084Python3 dk improve
MR2087Django 1.6
MR2088Add Content-Transfer-Encoding header to mailsender.
MR2093noc/noc#1008
MR2100Add search form from Maintenance
MR2102Allow - in git tags
MR2105User, Group ExtJS version
MR2108Add QSW-3470-28T-AC platform to Qtech. Switch get_version prefer to SNMP.
MR2110Django 1.7.11
MR2112noc/noc#914 Return first find profile that loader.
MR2115Add initial support for Extreme.Summit200 profile
MR2117Update apply-pools
MR2118Add depends on set_bi_id migration.
MR2122copy tags to clipboard
MR2125Cleanup Qtech.QSW2800.get_chassis_id for matcher use.
MR2132Update Angtel.Topaz profile
MR2137Add ingnore_errors param to http_get activator method.
MR2138Add config policy for IBM NOS
MR2139Add managed param to clickhouse model meta.
MR2142new profile - Polygon
MR2143Get serial number for Cisco ASR1000
MR2144Add noc user to docker container
MR2150Remove index field from clickhouse model.
MR2152HP Comware: platform matching, added getting serial number
MR2155Check column type when execute ch-migrate.
MR2157Added check of empty lines in the Object and Segment fields
MR2158Add get_inventory support for Eltex.MES profile
MR2161Move report metric to Report Detail format.
MR2162Docker with memcache
MR2168Set noc dir permissions
MR2178added OID for HP A3600-48-PoE
MR2180Merge noc-docs to main repo
MR2186aaa module
MR2187mongoengine 0.18
MR2188export filename as template appId_YYYYMMDDHHMMSS.csv
MR2189cachetools 3.1.1
MR2192tagfield add trigger copy to clipboard
MR2193define environmets
MR2194new/cm-notify
MR2195managedobject layout fix
MR2196ref book admin ExtJS version
MR2197docs: API autodocumentation
MR2198redirect after success login
MR2204Add administrative domain field to Report Latest Changes.
MR2206Add new Radio Metrics Row for rssi/cinr and rx/tx power metrics
MR2207Migrate to ComboTree
MR2213Add image options to script command.
MR2214Add config-violatile to Raisecom.ROS.
MR2217Add remote:deleted tag when managedobject removed from etl.
MR2218Update Ericsson SEOS Profile
MR2219Django 1.8
MR2222Check MR labels
MR2223documentation last releases description added
MR2224Mongo test hc
MR2225Reorganized vendor and profile documentation
MR2226Add Eltex.MES5448.get_inventory script
MR2227Add Eltex.DSLAM.get_inventory script
MR2229Add Eltex.LTP.get_inventory script
MR2233Add support for unpriveleged prompt for Eltex.MES5448
MR2235Django 1.9
MR2240Update Eltex.ESR profile
MR2244Add Eltex.LTE.get_inventory script
MR2246Service.use_mongo options to auto-connect to mongo database
MR2249Add Eltex.MA4000.get_inventory script
MR2250documentation historical releases description added
MR2252Build docs when merging to master
MR2253Replace yapf with black
MR2254Check changed fields when calculate datastream.
MR2256Docs config refactor
MR2257Django 1.10
MR2258Move tests to tmpfs
MR2260Add mirror options to gridvcs command.
MR2261Explicit MongoDB database connection
MR2262Django 1.11
MR2272Lower artifacts time
MR2273flake8: Disable E203 check
MR2280bandit analyzer check
MR2282Move custom mongoengine fields from noc.lib.nosql to noc.core.mongo.fields
MR2284caniusepython3 and pylint3k checks
MR2287flake8: black-friendly settings
MR2288Set discovery fatal error of profile do not detect profile on Generic devices.
MR2290Fix Eltex.MES5448.get_config script
MR2293Fix title and additional column to LinkDetailReport.
MR2297Django 1.11.22
MR2309docs: Tools documentation
MR2322black: Ignore deleted files
MR2323SA CLI/SNMP tests
MR2327Add Address column to Latest Changes report.
MR2336Fix column name in ReportLinkDetail.
MR2349Update DLink.DxS_Smart.init.py add DES-1210-52 v4
MR2351Update Huawei.VRP.get_version.py add CE platform.
MR2355Add DLink.DxS_Smart.get_capabilities for detecting of enabled LLDP protocol
MR2365Add hw_version and bootpromversion to ManagedObject BI models.
MR2376Add confdb support to Hikvision.DSKV8 profile.
MR2381Add confdb support to Beward.BD profile.
MR2395Add Eltex.WOP profile.
MR2404docs: ConfDB query language
MR2405Fix doc typo
MR2412MR: Check confdb label
MR2413Update NAG.SNR.get_arp.py
MR2416ConfDB: Fix Query doc
MR2421noc confdb syntax path parameter
MR2423docs: GA integration
MR2429Add confdb normalizer to Dahua.DH profile.
MR2434check status http query
MR2436CI: Disable test html report
MR2440install-packages: -v flag
MR2441Add connect() to some commands.
MR2442Extract collections to build docs
MR2449Update EdgeCore.ES profile
MR2450Add ConfDB normalizer to Cisco.IOS profile.
MR2455config: web.max_upload_size configuration parameter
MR2459login: register_last_login option
MR2460Update RU translation for Web services.
MR2461./noc confdb: tokenizer and normalizer helpers
MR2472Add confdb to managedobject card backend.
MR2475Use django-media package

Bugfixes

MRTitle
MR1847Fix DLink.DVG.get_chassis_id script
MR1952Fix SKS.SKS.get_interfaces script
MR2007Fix SKS.SKS.get_spanning_tree script
MR2008Fix Alstec.24xx.get_interfaces script
MR2009Fix detect Catalyst 4k platform
MR2010Fix path for release Dockerfiles
MR2017Fix Huawei.VRF.get_interfaces untagged from pvid.
MR2037Fix Generic.get_capabilities script when SNMP false.
MR2052Fix get_config scripts.
MR2056Fix TFortis.PSW get_interfaces.
MR2058Fix bulk update IPAM address usage cache.
MR2066ConfDB: NotMatch doesn't yield context if unresolved unbound variables left
MR2068Fix get_displayed_type method for clickhouse field.
MR2090Fix RouterOS parser
MR2097fix not ascii in description
MR2104Fix fix-pip
MR2116fix_metric_qtech_vendors
MR2119fix get_version script for old H3C devices
MR2127Fix upstream_connected_graph_template.
MR2128Fix Eltex.MES profile
MR2140Fix export inv.objectmodel to JSON
MR2151Fix Huawei.MA5600T pattern more.
MR2159Fix activator http_get params typo.
MR2167Fix add user in docker release image
MR2170Fix managed_object logger.
MR2173Fix ipv4 address validator
MR2175Fix inetrace speed attribute on ch dictionary
MR2177fixed H3C get_version for old devices like H3C S3100-8T-SI
MR2183Fix datasource interfaceattributes dictionary.
MR2191Fix Cisco.IOSXR.convert_interface_name
MR2199Fix caches
MR2203Fix telnet SB .. SE parsing
MR2205Fix command_disable_pager for NAG.SNR.
MR2208Fix reportmetrics field order.
MR2211Fix trace when convert UUID field on extdocapplication.
MR2216Fix LRUcache missing on etl chain
MR2230Fix typo in extdocapplication backend.
MR2237Fix trace when check type for new column
MR2247Profile: Compile syntax/operation error patterns as multiline
MR2255Fix do_pending_operations in selectorcache.
MR2259noc/noc#1047 Delete unused code from prefix delete_recursive.
MR2267Fix Huawei.MA5300.get_interfaces script
MR2268Fix Tag search query.
MR2269Replace XML namespace parameter on profile Hikvision.DSKV8.
MR2270Skip send_on_syntax_error when beef cli_error.
MR2274Fix prompt in Eltex.DSLAM profile
MR2275Fix backport label check
MR2276Fix Qtech.QOS.get_version script
MR2277Fix MikroTik.RouterOS.get_fqdn script
MR2278Fix MikroTik.RouterOS.get_cdp_neighbors script
MR2279Add operation_error in Cisco.IOS profile
MR2286Fix DCN.DCWL.get_interfaces profile for WL8200-TL1
MR2298Update Eltex.DSLAM profile
MR2302Fix Beward.BD more than 1 value trace.
MR2305Fix Caps model, Update caps when sync collections
MR2308Add yandex apikey configuration
MR2317Fix software_image option on script command.
MR2326Fix DCN.DCWL.get_interface_status_ex for WL8200.
MR2340Fix regex for duplicates packets in DCN.DCWL.ping script.
MR2341Fix refresh cfgping datastream when timepattern change
MR2343Fix DialPlan and NetworkSegment links in Project Card
MR2347Update NAG.SNR.get_interfaces.py add QSFP+
MR2348Update Huawei.VRP.get_portchannel.py add dynamic
MR2350Update DLink.DxS_Smart.get_portchannel.py - fix "type" output
MR2353simple report fixed
MR2356Update DLink.DxS_Smart.get_lldp_neighbors.py
MR2357reportmetrics fixed
MR2360Fix docker push command
MR2362Fix container column on ReportObjectDetail.
MR2364Return User.get_full_name() method, uses in ActiveAlarm backend.
MR2366Fix ReportLinkDetail when empty tags.
MR2367Add convert_interface_name to Iskratel.ESCOM.
MR2373fix_version_regex_eltex_mes
MR2375fix_get_config_MXA24
MR2377Fix Alcatel.TIMOS profile. Trace when iface has empty MAC.
MR2378Fix NSN.TIMOS.get_lldp_neighbors. Fix Multiline Remote PortsID and RemotePortDescription.
MR2379fix_get_config_Eltex_RG
MR2380Add empty values to Reports choices fields.
MR2386Add display omit to Juniper show configuration command
MR2387Fix Eltex.MES.get_version script on non stack devices
MR2388Fix deleted missing parameter on LRUCache.
MR2389Fix model cannot be resolved on managedobjectselector field.
MR2390Add set_unusable_password method to User model.
MR2393noc/noc#1042
MR2396#1068 Fix SQL and QTags broken by django upgrade
MR2397#1064 Migrate Handlers
MR2398#1066 Refactored CachedForeignKeyField
MR2399Fix Qtech.QSW8200.get_version regex.
MR2402Fix DCN.DCWL.get_interface_status_ex trace if not bss return.
MR2403Fix Alcatel.AOS.get_switchport untagged vlan list to int.
MR2407Fix AlarmEscalation wait_tt processing
MR2414ConfDB: Fix syntax glitches
MR2427Fix ./noc newapp
MR2432filter horizontal scroll fixed
MR2443Move _archive_db attribute from archive bi extractor to method.
MR2447Add configvalidation field on report_discovery result.
MR2448Fix Stream Audio config section on Hikvision.DSKV8.
MR2451ConfDB: Fix bound variables handling in NotMatch
MR2453users remove is_staff from model
MR2457Fix User's preferred language
MR2473Fix typo in Beward.BD normalizer.
MR2474Fix Huawei.VRP.get_version on S5600-HI platform

Code Cleanup

MRTitle
MR1885Migrate KB to ExtModelApplication
MR1977Model __str__ methods
MR2029Remove copy/paste mistake from Eltex.MES.get_config script
MR2034noc-model-str-cm-facts2
MR2035noc-model-str-core-ip
MR2036noc-model-str-services-lib
MR2049Test model's __str__
MR2050py3 dict .iteritems(), .itervalues(), .iterkeys()
MR20512to3 except fix
MR20542to3 exec fix
MR2060Remove urllib usage in profiles
MR2063test_base_parametrize. ver1
MR2064Wrap urllib/urllib2 usage
MR2067wrap StringIO
MR20742to3 import fix
MR20752to3 has_key fix
MR20762to3 fix urlparse
MR2080wrap maketrans call
MR20812to3 print fix
MR20822to3 fix map and filter
MR20832to3 imports fix
MR20852to3 itertools and itertools_import fixes
MR20862to3 long fix
MR20892to3 types fix
MR2094test_ecma48
MR2095test_ber
MR2101Remove south usage from BaseMigration class
MR2106Migration Runner, Bye-bye South
MR2107Cleanup models' init.py
MR2120fix_DeprecationWarning_main_0049_update_tags
MR2121fix_test_ip
MR2123test_crypto
MR2124test_mac
MR2133remove_total_seconds
MR2135test_matcher
MR2136test_validators
MR2221docs format mr tables
MR2236Fix docs path
MR2242Fix mongoengine imports
MR2243Fix mongoengine imports
MR2251Clean Huawei.MA5600T profile
MR22632to3: zip fix
MR2264Remove unused cm templates
MR22652to3: next fix
MR2266Fix ObjectId import
MR22712to3: dict fix
MR22832to3: xrange fix
MR2291black: aaa, bi, cm
MR2306black: crm, dev, dns, fixes, fm
MR2307black: gis, inv, ip, kb
MR2310fix_import_lib_nosql_part1
MR2311fix_import_lib_nosql_part2
MR2314black: migrations
MR2315fix_import_lib_nosql_part3
MR2316fix_import_lib_nosql_part4
MR2318black: sa
MR2319black: lib
MR2320black: core
MR2321black: Rest of stuff
MR2330Bump version
MR2354Add mongo connect to commands.
MR2358Speedup docker release images build with targeted builds
MR2406Drop unused fields
MR2411ConfDB syntax refactoring
MR2422Move profiles to profile.py
MR2424Fix get_version Infinet.Wanflex
MR2428Clean up noc.lib.nosql imports
MR2439Fix documentation glitches
MR2452Update Dynamic Dashboards.