NOC 19.3

16.08.2019

In accordance to our Release Policy we’re proudly present release 19.3.

19.3 release contains of 318 bugfixes, optimisations and improvements. Refer to the Release Notes for details.

Highlights

New Profiles

ConfDB Object and Interface Validation Policies

NOC 19.3 brings ability to validate object and interface settings using ConfDB Queries. This feature will became primary way of validation and will replace old CLIPS-based implementation in NOC 19.5.

It is good time to begin legacy validation policy migrations.

ConfDB Syntax Expansion

ConfDB got additional syntax for:

  • NTP support
  • Video, Audio settings and media streaming

ConfDB ‘raw’ Section

Raw tokenized input can be applied to ConfDB under the raw node. So platform-depended syntax can be processed via ConfDB Queries and validators natively, even if no platform-independent syntax exists. raw section may be enabled on Managed Object Profile level.

ConfDB ‘meta’ Section

NOC 19.3 adds meta section, containing valuable NOC database information directly exposed to ConfDB queries. Lots of additional information included:

  • Vendor
  • Platform
  • Software Version
  • Profile
  • Administrative Domain
  • Network Segment
  • Managed Object Tags
  • Managed Object Profile
  • Interface Profile
  • Interface Network Neighbors (i.e. Links)

ConfDB Normalizers

NOC 19.3 introduces 4 new profiles:

Uplink detection algorithm for RCA became configurable now. Following policies may be used:

  • Segment Hierarchy (previous algorithm)
  • Object Level
  • All Segment Objects
  • Lesser Management Address
  • Greater Management Address

Uplink policies may be configured at Network Segment Profile level. Multiple policies may be used with falling back to next policy until uplinks are detected.

Uplink policies are greatly improve the quality of topology-based RCA. See Uplink Policy for details.

Topology RCA Optimization

Topology-based Root-Cause analysis may be resource consumption. NOC 19.3 introduces new experimental accelerated mode called RCA Neighbor Cache. Smarter data precalculation and caching in combination of database query optimization and bulk updates allows to achieve 2-3 times speedup on real-world installations.

To enable the feature perform following steps:

  • Run fix:

$ noc fix apply fix_object_uplinks

Warning:

Alarm processing will be postponed when correlator process is stopped,
so alarm creation and clearing will be delayed until the correlator
process will be started again.

Prometheus Histograms

Prometheus histograms and quantiles may be exported via /metrics endpoint. Additional metrics may be enabled in config. See metrics section for details.

ObjectModel Tags

Inventory models got additional marking, which may be useful in various cases. See ObjectModel Tags for details. Model’s tags are also exposed into managedobject DataStream.

Django upgrade

Previous releases of NOC relied on venerable Django 1.4 dated back to 2012. Django’s team worked hard on improving their product according to their vision. Unfortunately they tend to introduce a lot of incompatibilities and upgrading to each next Django’s major release is the real pain. Django 1.4 fits our needs well but is not maintained and is incompatible with Python 3. So it is the time to to collect the pains.

We’d migrated from 1.4 to 1.5, then from 1.5 to 1.6, then followed by upgrades to 1.7, 1.8, 1.9, 1.10 and stopped at 1.11. During our stroll we’d became very disappointed by Django’s API stability and the high maintenance costs for the complex applications and applied some countermeasures.

NOC 19.3 brings following changes:

  • Django 1.11.22
  • Django’s auth contrib package has been replaced with AAA module.
  • South migrations has been replaced with our own Migration Engine.
  • All legacy Django admin applications (ModelApplication) has been replaced with ExtJS implementations.
  • Django will never create or modify database structure on its own (so-called syncdb).
  • Django static media repacked as django-media npkg package.

AAA module

User and Groups use NOC’s own implementation instead of Django’s ones. Besides the native ExtJS UI it allows future independent development according our needs. User Profile became the part of User model.

Migration Engine

South database migration engine stopped in development and users are urged to move to Django’s 1.7 built-in migration engine. During our investigations we’d found that we need to completely rewrite 500+ of existing migrations, migrations code will be bloated by the unnecessary abstractions and we need to invite the way to preserve old migration history.

So we’d developed migration engine, simple but powerful. Key benefits are:

  • Small, clean API.
  • Semi-automatical translation of existing migration.
  • Seamless migration history conversion.
  • Skipped migrations with from other development branch, may be applied later.

Development Process Changes

Code Formatting

NOC adopts black - the python code formatter. CI pipeline checks code formatting of changed python files. Any misformatting considered the error and CI pipeline fails at the [lint` stage. We recommend to add black formatting to git’s pre-commit hook or to the IDE’s on-save hook.

We’d already reformatted all ours codebase and NOC is now fully `PEP8](https://www.python.org/dev/peps/pep-0008/)-compatible. Docker container is also available. Use:

docker run --rm \
    -w /src \
    -v $PWD:/src \
    registry.getnoc.com/infrastructure/black:master \
    /usr/local/bin/black <file name>

to format file

Towards Python 3 compatibility

Python 3 compatibility became one of our priorities. With 19.3 we’d fixed lots of incompatibilities, upgraded same dependencies and becoming to get rid of unsupported ones. Though a lots of work and testing still required we’re expecting to reach full Python 3 compatibility in one of future releases.

MR Labels

We’re developed the policy for Merge Request’s (MR) labels. CI pipeline checks the labels and fails at the lint stage in case of errors. Label policy helps to organize testing and code reviewing process and quickly explains the goals of MR and subsystems affected.

Breaking Changes

Explicit MongoDB Connections

Prior to 19.3 NOC relied that importing of noc.lib.nosql automatically creates MongoDB connection. This kind of auto-magic used to work but requires to access all mongo-related stuff via noc.lib.nosql. Starting from 19.3 we’re beginning to cleanup API and the code and demand, that MongoDB connection is to be initialized implicitly.

For custom commands and python scripts

    from noc.core.mongo.connection import connect

    ...
    connect()

For custom services set service’s use_mongo property to True

Other Changes

  • ManagedObjectSelector.resolve_expression() renamed to ManagedObjectSelector.get_objects_from_expression()

New features

MR Title
!1682 ClickHouse LowCardinality support
!2091 New migrations framework
!2098 Migration loader, planner and tests
!2179 Prometheus histograms
!2181 ObjectModels tags field
!2190 RCA neighbor cache and accelerated topology correlation
!2220 Merge UserProfile into User model
!2228 Platform tags
!2245 ./noc test –idea-bookmarks option
!2285 Uplink Policy
!2372 Add MySQL Extractor
!2400 ConfDB: ntp support
!2418 #1077 ConfDB raw policy
!2419 Add new Profile Eltex.WOPLR
!2420 ConfDB: media section
!2426 ConfDB: Object Validation
!2433 ConfDB: meta section
!2438 ConfDB: Interface validation

Improvements

MR Title
!1888 Django 1.5
!1965 Add input_vlan_map and output_vlan_map fields to ConfDB syntax.
!2006 noc/noc#1032
!2020 Add autowidth column option to ReportLinkDetail.
!2021 Add autowidth column to ReportIfacesStatus.
!2023 Add autowidth column option to ReportAlarmDetail.
!2039 Add frozen first row in Detail Report.
!2041 Add subscribers profile filter to AlarmDetail Report.
!2048 Add ability to get vlans from bridge on MikroTik.RouterOS
!2055 Add ignoring snmp to profile checker.
!2065 ensure-indexes: Create index on fm.Uptime
!2078 add get_mac
!2084 Python3 dk improve
!2087 Django 1.6
!2088 Add Content-Transfer-Encoding header to mailsender.
!2093 noc/noc#1008
!2100 Add search form from Maintenance
!2102 Allow - in git tags
!2105 User, Group ExtJS version
!2108 Add QSW-3470-28T-AC platform to Qtech. Switch get_version prefer to SNMP.
!2110 Django 1.7.11
!2112 noc/noc#914 Return first find profile that loader.
!2115 Add initial support for Extreme.Summit200 profile
!2117 Update apply-pools
!2118 Add depends on set_bi_id migration.
!2122 copy tags to clipboard
!2125 Cleanup Qtech.QSW2800.get_chassis_id for matcher use.
!2132 Update Angtel.Topaz profile
!2137 Add ingnore_errors param to http_get activator method.
!2138 Add config policy for IBM NOS
!2139 Add managed param to clickhouse model meta.
!2142 new profile - Polygon
!2143 Get serial number for Cisco ASR1000
!2144 Add noc user to docker container
!2150 Remove index field from clickhouse model.
!2152 HP Comware: platform matching, added getting serial number
!2155 Check column type when execute ch-migrate.
!2157 Added check of empty lines in the Object and Segment fields
!2158 Add get_inventory support for Eltex.MES profile
!2161 Move report metric to Report Detail format.
!2162 Docker with memcache
!2168 Set noc dir permissions
!2178 added OID for HP A3600-48-PoE
!2180 Merge noc-docs to main repo
!2186 aaa module
!2187 mongoengine 0.18
!2188 export filename as template appId_YYYYMMDDHHMMSS.csv
!2189 cachetools 3.1.1
!2192 tagfield add trigger copy to clipboard
!2193 define environmets
!2194 new/cm-notify
!2195 managedobject layout fix
!2196 ref book admin ExtJS version
!2197 docs: API autodocumentation
!2198 redirect after success login
!2204 Add administrative domain field to Report Latest Changes.
!2206 Add new Radio Metrics Row for rssi/cinr and rx/tx power metrics
!2207 Migrate to ComboTree
!2213 Add image options to script command.
!2214 Add config-violatile to Raisecom.ROS.
!2217 Add remote:deleted tag when managedobject removed from etl.
!2218 Update Ericsson SEOS Profile
!2219 Django 1.8
!2222 Check MR labels
!2223 documentation last releases description added
!2224 Mongo test hc
!2225 Reorganized vendor and profile documentation
!2226 Add Eltex.MES5448.get_inventory script
!2227 Add Eltex.DSLAM.get_inventory script
!2229 Add Eltex.LTP.get_inventory script
!2233 Add support for unpriveleged prompt for Eltex.MES5448
!2235 Django 1.9
!2240 Update Eltex.ESR profile
!2244 Add Eltex.LTE.get_inventory script
!2246 Service.use_mongo options to auto-connect to mongo database
!2249 Add Eltex.MA4000.get_inventory script
!2250 documentation historical releases description added
!2252 Build docs when merging to master
!2253 Replace yapf with black
!2254 Check changed fields when calculate datastream.
!2256 Docs config refactor
!2257 Django 1.10
!2258 Move tests to tmpfs
!2260 Add mirror options to gridvcs command.
!2261 Explicit MongoDB database connection
!2262 Django 1.11
!2272 Lower artifacts time
!2273 flake8: Disable E203 check
!2280 bandit analyzer check
!2282 Move custom mongoengine fields from noc.lib.nosql to noc.core.mongo.fields
!2284 caniusepython3 and pylint3k checks
!2287 flake8: black-friendly settings
!2288 Set discovery fatal error of profile do not detect profile on Generic devices.
!2290 Fix Eltex.MES5448.get_config script
!2293 Fix title and additional column to LinkDetailReport.
!2297 Django 1.11.22
!2309 docs: Tools documentation
!2322 black: Ignore deleted files
!2323 SA CLI/SNMP tests
!2327 Add Address column to Latest Changes report.
!2336 Fix column name in ReportLinkDetail.
!2349 Update DLink.DxS_Smart.init.py add DES-1210-52 v4
!2351 Update Huawei.VRP.get_version.py add CE platform.
!2355 Add DLink.DxS_Smart.get_capabilities for detecting of enabled LLDP protocol
!2365 Add hw_version and bootpromversion to ManagedObject BI models.
!2376 Add confdb support to Hikvision.DSKV8 profile.
!2381 Add confdb support to Beward.BD profile.
!2395 Add Eltex.WOP profile.
!2404 docs: ConfDB query language
!2405 Fix doc typo
!2412 MR: Check confdb label
!2413 Update NAG.SNR.get_arp.py
!2416 ConfDB: Fix Query doc
!2421 noc confdb syntax path parameter
!2423 docs: GA integration
!2429 Add confdb normalizer to Dahua.DH profile.
!2434 check status http query
!2436 CI: Disable test html report
!2440 install-packages: -v flag
!2441 Add connect() to some commands.
!2442 Extract collections to build docs
!2449 Update EdgeCore.ES profile
!2450 Add ConfDB normalizer to Cisco.IOS profile.
!2455 config: web.max_upload_size configuration parameter
!2459 login: register_last_login option
!2460 Update RU translation for Web services.
!2461 ./noc confdb: tokenizer and normalizer helpers
!2472 Add confdb to managedobject card backend.
!2475 Use django-media package

Bugfixes

MR Title
!1847 Fix DLink.DVG.get_chassis_id script
!1952 Fix SKS.SKS.get_interfaces script
!2007 Fix SKS.SKS.get_spanning_tree script
!2008 Fix Alstec.24xx.get_interfaces script
!2009 Fix detect Catalyst 4k platform
!2010 Fix path for release Dockerfiles
!2017 Fix Huawei.VRF.get_interfaces untagged from pvid.
!2037 Fix Generic.get_capabilities script when SNMP false.
!2052 Fix get_config scripts.
!2056 Fix TFortis.PSW get_interfaces.
!2058 Fix bulk update IPAM address usage cache.
!2066 ConfDB: NotMatch doesn’t yield context if unresolved unbound variables left
!2068 Fix get_displayed_type method for clickhouse field.
!2090 Fix RouterOS parser
!2097 fix not ascii in description
!2104 Fix fix-pip
!2116 fix_metric_qtech_vendors
!2119 fix get_version script for old H3C devices
!2127 Fix upstream_connected_graph_template.
!2128 Fix Eltex.MES profile
!2140 Fix export inv.objectmodel to JSON
!2151 Fix Huawei.MA5600T pattern more.
!2159 Fix activator http_get params typo.
!2167 Fix add user in docker release image
!2170 Fix managed_object logger.
!2173 Fix ipv4 address validator
!2175 Fix inetrace speed attribute on ch dictionary
!2177 fixed H3C get_version for old devices like H3C S3100-8T-SI
!2183 Fix datasource interfaceattributes dictionary.
!2191 Fix Cisco.IOSXR.convert_interface_name
!2199 Fix caches
!2203 Fix telnet SB .. SE parsing
!2205 Fix command_disable_pager for NAG.SNR.
!2208 Fix reportmetrics field order.
!2211 Fix trace when convert UUID field on extdocapplication.
!2216 Fix LRUcache missing on etl chain
!2230 Fix typo in extdocapplication backend.
!2237 Fix trace when check type for new column
!2247 Profile: Compile syntax/operation error patterns as multiline
!2255 Fix do_pending_operations in selectorcache.
!2259 noc/noc#1047 Delete unused code from prefix delete_recursive.
!2267 Fix Huawei.MA5300.get_interfaces script
!2268 Fix Tag search query.
!2269 Replace XML namespace parameter on profile Hikvision.DSKV8.
!2270 Skip send_on_syntax_error when beef cli_error.
!2274 Fix prompt in Eltex.DSLAM profile
!2275 Fix backport label check
!2276 Fix Qtech.QOS.get_version script
!2277 Fix MikroTik.RouterOS.get_fqdn script
!2278 Fix MikroTik.RouterOS.get_cdp_neighbors script
!2279 Add operation_error in Cisco.IOS profile
!2286 Fix DCN.DCWL.get_interfaces profile for WL8200-TL1
!2298 Update Eltex.DSLAM profile
!2302 Fix Beward.BD more than 1 value trace.
!2305 Fix Caps model, Update caps when sync collections
!2308 Add yandex apikey configuration
!2317 Fix software_image option on script command.
!2326 Fix DCN.DCWL.get_interface_status_ex for WL8200.
!2340 Fix regex for duplicates packets in DCN.DCWL.ping script.
!2341 Fix refresh cfgping datastream when timepattern change
!2343 Fix DialPlan and NetworkSegment links in Project Card
!2347 Update NAG.SNR.get_interfaces.py add QSFP+
!2348 Update Huawei.VRP.get_portchannel.py add dynamic
!2350 Update DLink.DxS_Smart.get_portchannel.py - fix “type” output
!2353 simple report fixed
!2356 Update DLink.DxS_Smart.get_lldp_neighbors.py
!2357 reportmetrics fixed
!2360 Fix docker push command
!2362 Fix container column on ReportObjectDetail.
!2364 Return User.get_full_name() method, uses in ActiveAlarm backend.
!2366 Fix ReportLinkDetail when empty tags.
!2367 Add convert_interface_name to Iskratel.ESCOM.
!2373 fix_version_regex_eltex_mes
!2375 fix_get_config_MXA24
!2377 Fix Alcatel.TIMOS profile. Trace when iface has empty MAC.
!2378 Fix NSN.TIMOS.get_lldp_neighbors. Fix Multiline Remote PortsID and RemotePortDescription.
!2379 fix_get_config_Eltex_RG
!2380 Add empty values to Reports choices fields.
!2386 Add display omit to Juniper show configuration command
!2387 Fix Eltex.MES.get_version script on non stack devices
!2388 Fix deleted missing parameter on LRUCache.
!2389 Fix model cannot be resolved on managedobjectselector field.
!2390 Add set_unusable_password method to User model.
!2393 noc/noc#1042
!2396 #1068 Fix SQL and QTags broken by django upgrade
!2397 #1064 Migrate Handlers
!2398 #1066 Refactored CachedForeignKeyField
!2399 Fix Qtech.QSW8200.get_version regex.
!2402 Fix DCN.DCWL.get_interface_status_ex trace if not bss return.
!2403 Fix Alcatel.AOS.get_switchport untagged vlan list to int.
!2407 Fix AlarmEscalation wait_tt processing
!2414 ConfDB: Fix syntax glitches
!2427 Fix ./noc newapp
!2432 filter horizontal scroll fixed
!2443 Move _archive_db attribute from archive bi extractor to method.
!2447 Add configvalidation field on report_discovery result.
!2448 Fix Stream Audio config section on Hikvision.DSKV8.
!2451 ConfDB: Fix bound variables handling in NotMatch
!2453 users remove is_staff from model
!2457 Fix User’s preferred language
!2473 Fix typo in Beward.BD normalizer.
!2474 Fix Huawei.VRP.get_version on S5600-HI platform

Code Cleanup

MR Title
!1885 Migrate KB to ExtModelApplication
!1977 Model str methods
!2029 Remove copy/paste mistake from Eltex.MES.get_config script
!2034 noc-model-str-cm-facts2
!2035 noc-model-str-core-ip
!2036 noc-model-str-services-lib
!2049 Test model’s str
!2050 py3 dict .iteritems(), .itervalues(), .iterkeys()
!2051 2to3 except fix
!2054 2to3 exec fix
!2060 Remove urllib usage in profiles
!2063 test_base_parametrize. ver1
!2064 Wrap urllib/urllib2 usage
!2067 wrap StringIO
!2074 2to3 import fix
!2075 2to3 has_key fix
!2076 2to3 fix urlparse
!2080 wrap maketrans call
!2081 2to3 print fix
!2082 2to3 fix map and filter
!2083 2to3 imports fix
!2085 2to3 itertools and itertools_import fixes
!2086 2to3 long fix
!2089 2to3 types fix
!2094 test_ecma48
!2095 test_ber
!2101 Remove south usage from BaseMigration class
!2106 Migration Runner, Bye-bye South
!2107 Cleanup models’ init.py
!2120 fix_DeprecationWarning_main_0049_update_tags
!2121 fix_test_ip
!2123 test_crypto
!2124 test_mac
!2133 remove_total_seconds
!2135 test_matcher
!2136 test_validators
!2221 docs format mr tables
!2236 Fix docs path
!2242 Fix mongoengine imports
!2243 Fix mongoengine imports
!2251 Clean Huawei.MA5600T profile
!2263 2to3: zip fix
!2264 Remove unused cm templates
!2265 2to3: next fix
!2266 Fix ObjectId import
!2271 2to3: dict fix
!2283 2to3: xrange fix
!2291 black: aaa, bi, cm
!2306 black: crm, dev, dns, fixes, fm
!2307 black: gis, inv, ip, kb
!2310 fix_import_lib_nosql_part1
!2311 fix_import_lib_nosql_part2
!2314 black: migrations
!2315 fix_import_lib_nosql_part3
!2316 fix_import_lib_nosql_part4
!2318 black: sa
!2319 black: lib
!2320 black: core
!2321 black: Rest of stuff
!2330 Bump version
!2354 Add mongo connect to commands.
!2358 Speedup docker release images build with targeted builds
!2406 Drop unused fields
!2411 ConfDB syntax refactoring
!2422 Move profiles to profile.py
!2424 Fix get_version Infinet.Wanflex
!2428 Clean up noc.lib.nosql imports
!2439 Fix documentation glitches
!2452 Update Dynamic Dashboards.