Перейти к содержанию

NOC 19.2

In accordance to our Release Policy <releases-policy> we're proudly present release 19.2.

19.2 release contains of 337 bugfixes, optimisations and improvements.



19.2 introduces new configuration processing framework, the ConfDB. ConfDB is the crystallisation of our over decade experience of configuration processing. Set of carefully designed tools, clean API and flexible query language offer the unified way of configuration parsing, validation, processing and converting. Target area covers data extraction (without complex scripts), config validation, querying and inter-vendor configuration conversions.

We built ConfDB around idea of abstract intermediate config representation, somewhat similar with programming languages’ virtual machines, query language with Prolog-like logical programming capabilities, and common pipeline for config parsing.

Key features comparison against existing config processing models:

AreaOld ModelConfDB
Configuration fetchingVia get_config scriptget_config script, external storage download
Configuration storingGridVCSGridVCS with compression
Configuration parsingpyparsing parsers2-stage, tokenization and normalization
Parser complexityHightokenizers are reused between profiles, normalizers - easy
Parser performanceLowHigh
Config modelPlain list of predefined factstree-like structure
Custom factsAlmost impossibleEasy
Query language syntaxCLIPSPython
Query debuggingshellBuilt-in IDE
Discovery integrationNoneConfig-driven discovery checks

19.2 contains ConfDB support for EdgeCore.ES, Eltex.MES, Huawei.VRP, Juniper.JUNOS, MikroTik.RouterOS and Qtech.QSW2800 profiles.

Read more

Headless Mode

ConfDB allows the headless mode of operations, where “headless” means no direct access to equipment. Instead, configuration files are downloaded from external storage via FTP, SFTP or S3 API, stored to GridVCS and processed via ConfDB. Discovery can extract capabilities, interfaces and vlans directly from config. IPAM is also integrated with ConfDB, allowing to track vpns, address and prefixes usage.

Headed/Headless mode may be selected on per-device or per-profile basics via appropriate policies.

Headless mode is a valuable tool for due diligencies, security audits and forensics.

Config fetch policy

NOC got additional configuration to select, which configuration to prefer - startup or running. Config fetch policy may be applied on per-profile or per-object level.


IPAM got various improvements, including:

  • separate calculation of address and prefix usage
  • prefix profile’s setting to allow reuse of special addresses (network and broadcast)

Ethernet services

IGetInterfaces got the ES address family, allowing to describe complex vlan tag manipulations, often referred as Ethernet Services, Q-in-Q and so on.

GridVCS improvements

19.2 contains simple Mercurial’s mpatch implementation to read old deltas. So NOC is no longer depends on Mercurial.

New profiles

19.2 brings support for various IP-cams, including video-specific metrics. Hikvision.DSKV8, Dahua.DH, Axis.VAPIX profiles support lots of IP-cams out-of-the-box.

In addition, GWD.GFA profile has been added.


Profile checking engine (discovery’s profile check) has been refactored to separate class available both for discovery and for custom scripting.

DNS Zone expiration checks

19.2 restores DNS Zone expiration check, regressed long ago in microservices branch.

API Improvements

Script http client

Scripts’ http client has been reworked and got middleware support, allowing various wierd authentication schemes. Integration with web-only managed devices became easier.

nbi telemetry API

NBI service got a telemetry API <api-nbi-telemetry> for metrics pushing from remote agents, making the NOC full-blown telemetry collection and processing system for custom IoT applications.

Code quality


Code Quality always has been one of the major priorities. 19.2 contains lots of code cleaning, improving overall code quality and PEP8 compliance.


Testing infrastructure has been improved and we’d reached 100% test suite passing. Tests passing made the requirement for merge request acceptance.

Python3 compatibility

Python 2.7 will reach end-of-life at 2020-01-01, so we started complex porting project. Additional py3 compatibility linter has been added to the testing lint stage. Python3 compatibility became the requirement for merge request acceptance.

Additional py3 infrastructure for test suite running is ready and we’re working on 100% tests passing.

Breaking Changes


IGetConfig interface got additional policy input parameter. get_config scripts should recognize it and behave accordingly.

Possible values:

  • r - prefer running config
  • s - prefer startup config


Migration from 19.2 release is straightforward. Just deploy 19.2 or stable-19.2 tags from Tower.

New features

MR1568IGetInterfaces: ethernet services
MR1582Config tokenizers
MR1608#980 Config normalizers
MR1647ConfDB: Database and logic inference engine
MR1661Juniper.JUNOS: ConfDB tokenizer
MR1696Discovery: Headless config check
MR1706nbi: telemetry API
MR1776discovery: ConfDB methods
MR1783ConfDB: Applicators
MR1800Config Fetch Policy
MR1811ConfDB: query frontend
MR1837discovery: Running policy
MR1855Domain expiration check handlers
MR1872Bye-bye HG
MR1894#38 Add prefix_special_address_policy to Prefix Profile.
MR1926ProfileChecker class


MR1505fm.alarm: Check alarm is not already acknowledged/unacknowledged
MR1506Update AuditTrail funcional
MR1520Optimize IPv4 to integer conversion
MR1521Header panel: add last update button
MR1542Add support for ISCOM2608G
MR1545Report control initial commit
MR1550Patch uncol fm alarm sort
MR1551Update Managed Object profile form
MR1553Add 40G interfaces support to Cisco.IOS profile
MR1554ManagedObjectProfile: Refer to default caps profile by default
MR1558Update HP.ProCurve profile
MR1559fm-alarm i18n
MR1562Add hasAny and hasAll function to BI query language.
MR1564Add basic authentication to script.http methods.
MR1570Add initial support for GWD.GFA profile
MR1574Add pattern_unprivileged_prompt to Iskratel.MSAN.
MR1575Add http/https proto to script command.
MR1583Set enable datasource default settings to False.
MR1587pytest 4.1.1
MR1588HTTP Client auto JSON requests
MR1585Update GWD.GFA profile
MR1592Move FM reportalarmdetail application to ControlReport.
MR1594Add Interface Errors BIP metric to Huawei.MA5600T.
MR1596Script: http sessions, headers and cookies control
MR1609Add interval to PM scale function.
MR1611Add Hikvision.DSKV8 profile
MR1612Move isInMaintenance field on fm.alarm to bulk.
MR1613fm.alarm: alarm_class multi-select
MR1630combotree: reset pageSize limit
MR1634#980 noc.core.confdb package
MR1645Digest middleware
MR1655Update SKS.SKS profile
MR1648Huawei ma5600t
MR1660parse_table: Mixed tabs/space processing
MR1662NetworkSegment.get_nested_ids: Rewrite to fix out-of-memory condition
MR1663Update BDCOM.xPON profile
MR1664Add more d-link switches support
MR1666i18n: inv.reportifacestatus
MR1667Add more models to Eltex.MES profile
MR1669Add Dahua.DH profile for IP cam.
MR1671Add “port id subtype” to Huawei.VRP.get_lldp_neighbors.
MR1680header panel: hide last update, when don’t use
MR1681Add initial support for ZTE.ZXA10 profile
MR1684BEREncoder/BERDecoder singleton
MR1687BER: Opaque/float/double type decoders
MR1691Add patch attribute to Huawei.VRP.get_version.
MR1692Add CFP TransEth40G in verify-model.py for Finistar FTLQ8181EBLM
MR1693remote sync date
MR1694Add base implementation TCP RTSP client.
MR1695Catch RPC Error on STP discovery.
MR1698Add capabilities to script matchers.
MR1705Add BRAS capabilities to Juniper.JUNOS profile
MR1711ClickHouse client: Allow empty passwords
MR1716Add group by slot to `Memory
MR1721MikroTik.RouterOS: Match multi-line prompts
MR1733Add matchers to Alstec.24xx.get_metrics.
MR1737Update ZTE.ZXA10 profile
MR1738discovery: Split config fetching and config validation stages
MR1742Add state decorator to ip.vrf API.
MR1743Add ack_ts, ack_user field to Alarms BI model
MR1750Add serials number field to managed_object BI models and extractor.
MR1753Check handler permission before call.
MR1758Add ManagedObjectAttribute to card template.
MR1761Add initial support for Axis.VAPIX profile
MR1762Add Dahua.DH.get_local_version script.
MR1763Add Beward.BD.get_local_users script.
MR1767ConfDB: Customized syntax
MR1774Profile’s confdb package
MR1775Update Ericsson SEOS profile
MR1779Migrate NBI loader to BaseLoader.
MR1781Normalizer: defer() and RouterOS normalizer skeleton
MR1784Add datastream meta field to ensure-indexes command.
MR1791Update Juniper.JUNOS.get_interfaces script to support tunnels
MR1794Add operation error in Huawei.MA5600T profile
MR1796Do not expand tabs in parse_table() on some rotten devices
MR1814ConfDB: VRF configuration
MR1820Eltex.MES: MES-2324B/MES-2348B platforms
MR1822ConfDB: RebaseApplicator
MR1833Update get_portchannel.py
MR1835Show service’s startup time
MR1836discovery: Load ConfDB on demand even if config check is disabled
MR1858Nocmc integration
MR1859add lldp zyxel interface
MR1860External Storage: Check directory is not exists before creation
MR1865Add Jiniper.JunOS confdb normalizer.
MR1868ConfDB tests
MR1877Split docker-compose projects
MR1878Profile: Show DeprecationWarning when profile is not in profile.py
MR1879Optimize custom field loading
MR1881pymongo 3.8, mongoengine 0.17.0
MR1883Add mongodb rs init info. Pretty hacky
MR1884Add future 2 stage. Make it mandatory
MR1887Add Address Usage column to IPAM.
MR1896VC.VLAN: add Description column, add interface name to L3 interfaces list
MR1907Add future to docker requirements.
MR1910test: Report empty module’s docstrings
MR1911#996 Add project field to Subscriber and Supplier.
MR1913Exclude self when AddressRange overlaps check when save.
MR1914Getting vrf_id when confdb interface discovery.
MR1916ConfDB: vpn-id
MR1919Add type field to Qtech.QSW2800 memory_usage metric.
MR1921Deny move cpestatus to anothet managed_object when it inactive
MR1922Clean etl mapping key when load.
MR1923Qtech.QSW2800 more more patterns
MR1924Replace generic profile getting method in report_objectstat.
MR1925Reorganize model tests
MR1927Add support for SWA-2200-16P-AC device
MR1935Add Patch Version column and autowidth option to Object Detail Repport.
MR1936Remove volumes also when down tests
MR1949Add export_beef parameter to script.
MR1955Add Eltex.MES confdb normalizer.
MR1959Move MongoCache index creation into ensure-indexes
MR1960Replace ensure_index with create_index
MR1964.to_json() and .get_json_path() tests
MR1978ipam: Policy defaults in “New Prefix” form
MR1992Add SNMP method to Huawei.MA5600T.get_metrics.
MR1995#1030 Add DataSourceCache to ensure-index.


MR1495Fix Rotek.RTBSv1 cli errors
MR1496Fix Rotek.RTBS cli errors
MR1504Fix interfaceattributes datasource filter.
MR1509Fix Qtech.QSW.get_switchport script
MR1511Fix match trace on SKS.SKS profile.
MR1513Fix MACDB circular references.
MR1517Fix is_builtin field
MR1519Fix threshold profile label in managed object profile metrics view.
MR1524Fix Eltex.MES Lldp by snmp
MR1526#998 Send bi_id field to frontend as string.
MR1528Fix noc script command. Clean update-spec collect command.
MR1530Fix TFortis.get_version script.
MR1531Revert DLink.DxS.get_spanning_tree changes.
MR1533Fix Alcatel.TIMOS.get_interfaces empty mac.
MR1536Fix translation command BI model path
MR1540Fm alarm fix
MR1541localization fix (min)
MR1543Fix Extreme.XOS.get_lldp_neighbors script
MR1544Fix Detail Alarm report
MR1546Fix convert interface name for Qtech.QSW
MR1547Fix Juniper.JUNOS.get_mac_address_table script
MR1548Fix clean thresholdprofile migration
MR1549Fix get_lldp_neighbors script for Juniper.Junos
MR1552Fix Cisco.ASA parse config
MR1555Fix Time Pattern
MR1566Fix DCN.DCWL.get_interfaces typo.
MR1569fm-alarm save filters in local store
MR1571Fix Qtech.QSW profile for lldp
MR1576Fix Control translation.
MR1577Fix Siklu.EH.get_lldp_neighbors.
MR1586Fix Alcatel.AOS.get_switchport script
MR1597#1006 Fix wipe user
MR1598Fix detect snmp v1 for profile check rule.
MR1601Fix error in reporting ifSpeed on some D-Link devices
MR1603Fix syslog CH Model message field description.
MR1610#1009 Fix Add/Insert object in inventory tree.
MR1614fm.alarm: view recent alarms fixed
MR1615Fix Raisecom.ROS.get_interfaces trace.
MR1617Add metrics to selfmon fm collector.
MR1619Fix Huawei.MA5600T.get_interfaces get vlans.
MR1623Fix Raisecom.ROS.get_interfaces script
MR1624Fix DLink.DxS_Smart.get_lldp_neighbors script
MR1625Fix Eltex.MES5448.get_version script
MR1626Fix Interfaces status ex for physical interfaces
MR1627use ExtJS .endsWith for IE compatibility
MR1632Some fixes for Huawei.MA5600T profile.
MR1636Fix import
MR1638Fix Eltex.MES.get_spanning_tree script
MR1639Fix for Oob and Stack-port interfaces
MR1641#990 grid.column, override formatter to ‘htmlEncode’
MR1642Fix Eltex.MES.get_interfaces script
MR1643Fix Cisco.IOSXR.get_inventory script
MR1644Fix discovery cpe
MR1649Change replace character for Grafana PM Dashboard
MR1652Add fix_cpe_header function to Huawei.MA5600T profile.
MR1654grid.column, override formatter fix
MR1656Fix DLink.DxS_Industrial_CLI interface naming
MR1657Fix DLink.DxS.get_mac_address_table script
MR1670Fix snmp_metrics_get_chunk on Rotek.BT profile.
MR1677Fix stupid copy/paste error
MR1683Config MUST NOT match re validation policy
MR1689Fix DCN.DCWL.get_metrics scale and type Interface metrics.
MR1697fix juniper.junos metric Subscribers (for version >16) and Temperature
MR1700Fix Raisecom.ROS.get_inventory trace when Unknown SFP.
MR1702Fix Siklu.EH.get_lldp_neighbors empty section trace.
MR1712Fix Rotek RTBSv1 profile
MR1718Fix CPU, Memory, Temperature metrics for Huawei.VRP
MR1722Some fixes on IP Cam profiles Dahua.DH, Beward.BD, Hikvision.DSKV8
MR1723juniper.junos metric Subscribers without caps
MR1726Huawei.MA5600T move profile settings to profile.py
MR1728juniper.junos metric Subscribers without caps
MR1731Alstec.24xx.get_metrics fix trace when no controller module.
MR1734Fix matchers when check string in iter.
MR1735Fix clickhouse backend IPv4 displayed field.
MR1741#977 Fix collect SLA metrics in Cisco.IOS.get_metrics.
MR1745Fix Huawei.MA5600T.get_interfaces script
MR1747#939 Fix Huawei.VPR profile system patter_prompt
MR1749Fix chwriter channels_active metric trace.
MR1751#973 Fix managed_object two in card path exception.
MR1752Fix CursorTimeout when execute datastream rebuild command on large collection.
MR1754Fix previous commit
MR1755Fix Cisco.IOS.get_interfaces script
MR1756Fix Orion.NOS profile
MR1757Fix Juniper.JUNOS.get_version script
MR1760Fix fm models. add json_unique_fields
MR1780Fix exception in DLink.DxS_Smart profile
MR1788Fix metrics in card
MR1789Fix MikroTik.RouterOS.get_interfaces script
MR1793Fix low privelegies error in DLink.DxS profile
MR1802Fix box.job typo.
MR1805Fix Cisco.NXOS.get_inventory script
MR1817Fix DLink.DxS clean_lldp_neighbor() function
MR1838Fix Eltex.MES.get_lldp_neighbors script
MR1839Fix MikroTik.RouterOS.get_lldp_neighbors script
MR1840Fix Eltex.RG.get_version script
MR1841Fix Eltex.DSLAM profile
MR1842Fix Hiawei.MA5300. Disable session use.
MR1844Fix Qtech.QSW2500 script
MR1852Fix ManagedObject card not shows interfaces without metrics.
MR1857#1014 Fix ‘Config MUST NOT match string’ validator
MR1861Fix row_limit in web app.
MR1864Fix ch-policy dot name on command.
MR1867Fix ManagedObject vendor migration.
MR1869Fix wrong scripts name
MR1871Fix tests
MR1874py3: Fix threadpool
MR1875fix get_config tplink t2600 confdb
MR1876Fix tests
MR1889Fix MikroTik.RouterOS profile
MR1890Fix SKS.SKS.get_spanning_tree script
MR1897fix get_inv for X650
MR1899Fix to_python method for Clickhouse nested field.
MR1901Fix SKS.SKS convert_interface_name() function
MR1902Fix SKS.SKS.get_spanning_tree script
MR1903Fix Iskratel.ESCOM.get_spanning_tree script
MR1905Fix Nateks.FlexGain.get_interfaces script
MR1908Fix limit check on extapplication.
MR1928Fix DLink.DxS.get_interface_status_ex script
MR1929Revert “Merge branch ‘qsw2800-more-patterns’ into ‘master’”
MR1930Fix Alstec.24xx.get_interfaces script
MR1931Catch SNMP Error when Huawei.VRP SNMP worked.
MR1938Fix classifier rule cleanup code.
MR1939Some fixes for Hikvision.DSKV8 profile.
MR1940Fix get_handler error message typo.
MR1942Fix CPEStatus discovery when calculate diff for new cpe.
MR1943Fix Huawei.MA5600T pattern more for Honet.
MR1946Fix multiple jobs on master branch conflict. Prettify a little bit.
MR1950Fix Huawei.VRP.get_capabilities snmp check when get.
MR1954Some fixes in Generic snmp scripts.
MR1961Fix choices attribute in StringList interface parameter.
MR1966Fix Juniper.JunOS confdb normalizer.
MR1967Fix job_handler validator in Workflow editor backend.
MR1979Fix Cisco.IOS.get_version script
MR1986LineTokenizer: Fix string_quote processing
MR1987Fix Qtech.QSW2800.get_inventory datetime import.
MR1989Fix help command execute on custom.
MR1991Fix get_config script on profiles for policy.
MR1993#1041 Fix segment update_summary fix.
MR1994#1039 Make IPAM VRF search ungrouped VRF.
MR1996Fix calculate address and prefix usage.
MR2001Fix reportdatasource backend.
MR2003#1044 Fix import error on reportoverlappedoids.
MR2004#981 Fix Cisco.ASA pattern_username match to Last login message.

Code cleanup

MR1591Remove wildcard imports
MR1604DocInline code cleanup
MR1605Remove inv and fm monitor deprecated API
MR1606Remove cocomo report
MR1792Set yapf column limit to 120
MR1797flake8: Disable W605
MR1808cm_migrations. code refactoring
MR1809crm_migrations. code refactoring
MR1810dns_migrations. code refactoring
MR1812fm_migrations. code refactoring
MR1813gis_migrations. code refactoring
MR1815inv_migrations. code refactoring.
MR1818ip_migrations. code refactoring
MR1819kb_migrations. code refactoring
MR1821main_migrations. code refactoring
MR1823peer_migrations. code refactoring
MR1824phone_migrations. code refactoring
MR1825pm_migrations. code refactoring
MR1826project_migrations. code refactoring
MR1828sa_migrations. code refactoring
MR1829sla_migrations. code refactoring
MR1830vc_migrations. code refactoring
MR1831wf_migrations. code refactoring.
MR1832Update init.py
MR1850flake8: Fix tests/
MR1851Remove lib/tt.py
MR1856Remove obsolete periodics
MR1870Cleanup noc.settings import.
MR1880py3: Rename izip_longest
MR1882Cleanup init.py
MR1891flake8_E2_E3_ver2 without sa_profiles, services, tests
MR1892fix_migrations. DeprecationWarning pymongo
MR1898flake8_E2_E3_ver2. services
MR1909flake8_E2_E3_ver2. sa_profiles
MR1915Fix @on_delete_check decorators
MR1918Code cleanup
MR1958Cleanup DeprecationWarnings
MR1962Remove socketKeepAlive option
MR1976lib.app cleanup
MR19802to3 idioms
MR19822to3 basestring
MR19832to3 set literal
MR1984Port legacy tests
MR19852to3 metaclass