NOC 19.2

12.05.2019

In accordance to our Release Policy we’re proudly present release 19.2.

19.2 release contains of 337 bugfixes, optimisations and improvements.

Highlights

ConfDB

19.2 introduces new configuration processing framework, the ConfDB. ConfDB is the crystallisation of our over decade experience of configuration processing. Set of carefully designed tools, clean API and flexible query language offer the unified way of configuration parsing, validation, processing and converting. Target area covers data extraction (without complex scripts), config validation, querying and inter-vendor configuration conversions.

We built ConfDB around idea of abstract intermediate config representation, somewhat similar with programming languages’ virtual machines, query language with Prolog-like logical programming capabilities, and common pipeline for config parsing.

Key features comparison against existing config processing models:

Area Old Model ConfDB
Configuration fetching Via get_config script get_config script, external storage download
Configuration storing GridVCS GridVCS with compression
Configuration parsing pyparsing parsers 2-stage, tokenization and normalization
Parser complexity High tokenizers are reused between profiles, normalizers - easy
Parser performance Low High
Config model Plain list of predefined facts tree-like structure
Custom facts Almost impossible Easy
Query language syntax CLIPS Python
Query debugging shell Built-in IDE
Discovery integration None Config-driven discovery checks

19.2 contains ConfDB support for EdgeCore.ES, Eltex.MES, Huawei.VRP, Juniper.JUNOS, MikroTik.RouterOS and Qtech.QSW2800 profiles.

Read more …

Headless Mode

ConfDB allows the headless mode of operations, where “headless” means no direct access to equipment. Instead, configuration files are downloaded from external storage via FTP, SFTP or S3 API, stored to GridVCS and processed via ConfDB. Discovery can extract capabilities, interfaces and vlans directly from config. IPAM is also integrated with ConfDB, allowing to track vpns, address and prefixes usage.

Headed/Headless mode may be selected on per-device or per-profile basics via appropriate policies.

Headless mode is a valuable tool for due diligences, security audits and forensics.

Config fetch policy

NOC got additional configuration to select, which configuration to prefer - startup or running. Config fetch policy may be applied on per-profile or per-object level.

IPAM

IPAM got various improvements, including: * separate calculation of address and prefix usage * prefix profile’s setting to allow reuse of special addresses (network and broadcast)

Ethernet services

IGetInterfaces got the ES address family, allowing to describe complex vlan tag manipulations, often referred as Ethernet Services, Q-in-Q and so on.

GridVCS improvements

19.2 contains simple Mercurial’s mpatch implementation to read old deltas. So NOC is no longer depends on Mercurial.

New profiles

19.2 brings support for various IP-cams, including video-specific metrics. Hikvision.DSKV8, Dahua.DH, Axis.VAPIX profiles support lots of IP-cams out-of-the-box.

In addition, GWD.GFA profile has been added.

ProfileChecker

Profile checking engine (discovery’s profile check) has been refactored to separate class available both for discovery and for custom scripting.

DNS Zone expiration checks

19.2 restores DNS Zone expiration check, regressed long ago in microservices branch.

API Improvements

Script http client

Scripts’ http client has been reworked and got middleware support, allowing various wierd authentication schemes. Integration with web-only managed devices became easier.

nbi telemetry API

NBI service got a telemetry API for metrics pushing from remote agents, making the NOC full-blown telemetry collection and processing system for custom IoT applications.

Code quality

PEP8

Code Quality always has been one of the major priorities. 19.2 contains lots of code cleaning, improving overall code quality and PEP8 compliance.

Tests

Testing infrastructure has been improved and we’d reached 100% test suite passing. Tests passing made the requirement for merge request acceptance.

Python3 compatibility

Python 2.7 will reach end-of-life at 2020-01-01, so we started complex porting project. Additional py3 compatibility linter has been added to the testing lint stage. Python3 compatibility became the requirement for merge request acceptance.

Additional py3 infrastructure for test suite running is ready and we’re working on 100% tests passing.

Breaking Changes

IGetConfig

IGetConfig interface got additional policy input parameter. get_config scripts should recognize it and behave accordingly.

Possible values:

  • r - prefer running config
  • s - prefer startup config

Migration

Migration from 19.2 release is straightforward. Just deploy 19.2 or stable-19.2 tags from Tower.

New features

MR Title
!1568 IGetInterfaces: ethernet services
!1582 Config tokenizers
!1608 #980 Config normalizers
!1647 ConfDB: Database and logic inference engine
!1661 Juniper.JUNOS: ConfDB tokenizer
!1696 Discovery: Headless config check
!1706 nbi: telemetry API
!1776 discovery: ConfDB methods
!1783 ConfDB: Applicators
!1800 Config Fetch Policy
!1811 ConfDB: query frontend
!1837 discovery: Running policy
!1855 Domain expiration check handlers
!1872 Bye-bye HG
!1894 #38 Add prefix_special_address_policy to Prefix Profile.
!1926 ProfileChecker class

Improvements

MR Title
!1505 fm.alarm: Check alarm is not already acknowledged/unacknowledged
!1506 Update AuditTrail funcional
!1520 Optimize IPv4 to integer conversion
!1521 Header panel: add last update button
!1542 Add support for ISCOM2608G
!1545 Report control initial commit
!1550 Patch uncol fm alarm sort
!1551 Update Managed Object profile form
!1553 Add 40G interfaces support to Cisco.IOS profile
!1554 ManagedObjectProfile: Refer to default caps profile by default
!1558 Update HP.ProCurve profile
!1559 fm-alarm i18n
!1562 Add hasAny and hasAll function to BI query language.
!1564 Add basic authentication to script.http methods.
!1570 Add initial support for GWD.GFA profile
!1574 Add pattern_unprivileged_prompt to Iskratel.MSAN.
!1575 Add http/https proto to script command.
!1583 Set enable datasource default settings to False.
!1587 pytest 4.1.1
!1588 HTTP Client auto JSON requests
!1585 Update GWD.GFA profile
!1592 Move FM reportalarmdetail application to ControlReport.
!1594 Add Interface Errors BIP metric to Huawei.MA5600T.
!1596 Script: http sessions, headers and cookies control
!1609 Add interval to PM scale function.
!1611 Add Hikvision.DSKV8 profile
!1612 Move isInMaintenance field on fm.alarm to bulk.
!1613 fm.alarm: alarm_class multi-select
!1630 combotree: reset pageSize limit
!1634 #980 noc.core.confdb package
!1645 Digest middleware
!1655 Update SKS.SKS profile
!1648 Huawei ma5600t
!1660 parse_table: Mixed tabs/space processing
!1662 NetworkSegment.get_nested_ids: Rewrite to fix out-of-memory condition
!1663 Update BDCOM.xPON profile
!1664 Add more d-link switches support
!1666 i18n: inv.reportifacestatus
!1667 Add more models to Eltex.MES profile
!1669 Add Dahua.DH profile for IP cam.
!1671 Add “port id subtype” to Huawei.VRP.get_lldp_neighbors.
!1680 header panel: hide last update, when don’t use
!1681 Add initial support for ZTE.ZXA10 profile
!1684 BEREncoder/BERDecoder singleton
!1687 BER: Opaque/float/double type decoders
!1691 Add patch attribute to Huawei.VRP.get_version.
!1692 Add CFP TransEth40G in verify-model.py for Finistar FTLQ8181EBLM
!1693 remote sync date
!1694 Add base implementation TCP RTSP client.
!1695 Catch RPC Error on STP discovery.
!1698 Add capabilities to script matchers.
!1705 Add BRAS capabilities to Juniper.JUNOS profile
!1711 ClickHouse client: Allow empty passwords
!1716 Add group by slot to `Memory
!1721 MikroTik.RouterOS: Match multi-line prompts
!1733 Add matchers to Alstec.24xx.get_metrics.
!1737 Update ZTE.ZXA10 profile
!1738 discovery: Split config fetching and config validation stages
!1742 Add state decorator to ip.vrf API.
!1743 Add ack_ts, ack_user field to Alarms BI model
!1750 Add serials number field to managed_object BI models and extractor.
!1753 Check handler permission before call.
!1758 Add ManagedObjectAttribute to card template.
!1761 Add initial support for Axis.VAPIX profile
!1762 Add Dahua.DH.get_local_version script.
!1763 Add Beward.BD.get_local_users script.
!1767 ConfDB: Customized syntax
!1774 Profile’s confdb package
!1775 Update Ericsson SEOS profile
!1779 Migrate NBI loader to BaseLoader.
!1781 Normalizer: defer() and RouterOS normalizer skeleton
!1784 Add datastream meta field to ensure-indexes command.
!1791 Update Juniper.JUNOS.get_interfaces script to support tunnels
!1794 Add operation error in Huawei.MA5600T profile
!1796 Do not expand tabs in parse_table() on some rotten devices
!1814 ConfDB: VRF configuration
!1820 Eltex.MES: MES-2324B/MES-2348B platforms
!1822 ConfDB: RebaseApplicator
!1833 Update get_portchannel.py
!1835 Show service’s startup time
!1836 discovery: Load ConfDB on demand even if config check is disabled
!1858 Nocmc integration
!1859 add lldp zyxel interface
!1860 External Storage: Check directory is not exists before creation
!1865 Add Jiniper.JunOS confdb normalizer.
!1868 ConfDB tests
!1873 Docker CODEOWNERS
!1877 Split docker-compose projects
!1878 Profile: Show DeprecationWarning when profile is not in profile.py
!1879 Optimize custom field loading
!1881 pymongo 3.8, mongoengine 0.17.0
!1883 Add mongodb rs init info. Pretty hacky
!1884 Add future 2 stage. Make it mandatory
!1887 Add Address Usage column to IPAM.
!1896 VC.VLAN: add Description column, add interface name to L3 interfaces list
!1907 Add future to docker requirements.
!1910 test: Report empty module’s docstrings
!1911 #996 Add project field to Subscriber and Supplier.
!1913 Exclude self when AddressRange overlaps check when save.
!1914 Getting vrf_id when confdb interface discovery.
!1916 ConfDB: vpn-id
!1919 Add type field to Qtech.QSW2800 memory_usage metric.
!1921 Deny move cpestatus to anothet managed_object when it inactive
!1922 Clean etl mapping key when load.
!1923 Qtech.QSW2800 more more patterns
!1924 Replace generic profile getting method in report_objectstat.
!1925 Reorganize model tests
!1927 Add support for SWA-2200-16P-AC device
!1935 Add Patch Version column and autowidth option to Object Detail Repport.
!1936 Remove volumes also when down tests
!1949 Add export_beef parameter to script.
!1955 Add Eltex.MES confdb normalizer.
!1959 Move MongoCache index creation into ensure-indexes
!1960 Replace ensure_index with create_index
!1964 .to_json() and .get_json_path() tests
!1978 ipam: Policy defaults in “New Prefix” form
!1992 Add SNMP method to Huawei.MA5600T.get_metrics.
!1995 #1030 Add DataSourceCache to ensure-index.

Bugfixes

MR Title
!1495 Fix Rotek.RTBSv1 cli errors
!1496 Fix Rotek.RTBS cli errors
!1504 Fix interfaceattributes datasource filter.
!1509 Fix Qtech.QSW.get_switchport script
!1511 Fix match trace on SKS.SKS profile.
!1513 Fix MACDB circular references.
!1517 Fix is_builtin field
!1519 Fix threshold profile label in managed object profile metrics view.
!1524 Fix Eltex.MES Lldp by snmp
!1526 #998 Send bi_id field to frontend as string.
!1528 Fix noc script command. Clean update-spec collect command.
!1530 Fix TFortis.get_version script.
!1531 Revert DLink.DxS.get_spanning_tree changes.
!1533 Fix Alcatel.TIMOS.get_interfaces empty mac.
!1536 Fix translation command BI model path
!1540 Fm alarm fix
!1541 localization fix (min)
!1543 Fix Extreme.XOS.get_lldp_neighbors script
!1544 Fix Detail Alarm report
!1546 Fix convert interface name for Qtech.QSW
!1547 Fix Juniper.JUNOS.get_mac_address_table script
!1548 Fix clean thresholdprofile migration
!1549 Fix get_lldp_neighbors script for Juniper.Junos
!1552 Fix Cisco.ASA parse config
!1555 Fix Time Pattern
!1557 Cisco.SMB
!1566 Fix DCN.DCWL.get_interfaces typo.
!1569 fm-alarm save filters in local store
!1571 Fix Qtech.QSW profile for lldp
!1576 Fix Control translation.
!1577 Fix Siklu.EH.get_lldp_neighbors.
!1586 Fix Alcatel.AOS.get_switchport script
!1597 #1006 Fix wipe user
!1598 Fix detect snmp v1 for profile check rule.
!1601 Fix error in reporting ifSpeed on some D-Link devices
!1603 Fix syslog CH Model message field description.
!1610 #1009 Fix Add/Insert object in inventory tree.
!1614 fm.alarm: view recent alarms fixed
!1615 Fix Raisecom.ROS.get_interfaces trace.
!1617 Add metrics to selfmon fm collector.
!1619 Fix Huawei.MA5600T.get_interfaces get vlans.
!1623 Fix Raisecom.ROS.get_interfaces script
!1624 Fix DLink.DxS_Smart.get_lldp_neighbors script
!1625 Fix Eltex.MES5448.get_version script
!1626 Fix Interfaces status ex for physical interfaces
!1627 use ExtJS .endsWith for IE compatibility
!1632 Some fixes for Huawei.MA5600T profile.
!1636 Fix import
!1637 beward_profile
!1638 Fix Eltex.MES.get_spanning_tree script
!1639 Fix for Oob and Stack-port interfaces
!1641 #990 grid.column, override formatter to ‘htmlEncode’
!1642 Fix Eltex.MES.get_interfaces script
!1643 Fix Cisco.IOSXR.get_inventory script
!1644 Fix discovery cpe
!1649 Change replace character for Grafana PM Dashboard
!1652 Add fix_cpe_header function to Huawei.MA5600T profile.
!1654 grid.column, override formatter fix
!1656 Fix DLink.DxS_Industrial_CLI interface naming
!1657 Fix DLink.DxS.get_mac_address_table script
!1658 sova/rtk#671
!1659 #1018
!1670 Fix snmp_metrics_get_chunk on Rotek.BT profile.
!1677 Fix stupid copy/paste error
!1683 Config MUST NOT match re validation policy
!1689 Fix DCN.DCWL.get_metrics scale and type Interface metrics.
!1697 fix juniper.junos metric Subscribers (for version >16) and Temperature
!1700 Fix Raisecom.ROS.get_inventory trace when Unknown SFP.
!1702 Fix Siklu.EH.get_lldp_neighbors empty section trace.
!1712 Fix Rotek RTBSv1 profile
!1718 Fix CPU, Memory, Temperature metrics for Huawei.VRP
!1722 Some fixes on IP Cam profiles Dahua.DH, Beward.BD, Hikvision.DSKV8
!1723 juniper.junos metric Subscribers without caps
!1726 Huawei.MA5600T move profile settings to profile.py
!1728 juniper.junos metric Subscribers without caps
!1731 Alstec.24xx.get_metrics fix trace when no controller module.
!1734 Fix matchers when check string in iter.
!1735 Fix clickhouse backend IPv4 displayed field.
!1741 #977 Fix collect SLA metrics in Cisco.IOS.get_metrics.
!1745 Fix Huawei.MA5600T.get_interfaces script
!1747 #939 Fix Huawei.VPR profile system patter_prompt
!1749 Fix chwriter channels_active metric trace.
!1751 #973 Fix managed_object two in card path exception.
!1752 Fix CursorTimeout when execute datastream rebuild command on large collection.
!1754 Fix previous commit
!1755 Fix Cisco.IOS.get_interfaces script
!1756 Fix Orion.NOS profile
!1757 Fix Juniper.JUNOS.get_version script
!1760 Fix fm models. add json_unique_fields
!1780 Fix exception in DLink.DxS_Smart profile
!1788 Fix metrics in card
!1789 Fix MikroTik.RouterOS.get_interfaces script
!1793 Fix low privelegies error in DLink.DxS profile
!1799 fix
!1802 Fix box.job typo.
!1805 Fix Cisco.NXOS.get_inventory script
!1817 Fix DLink.DxS clean_lldp_neighbor() function
!1838 Fix Eltex.MES.get_lldp_neighbors script
!1839 Fix MikroTik.RouterOS.get_lldp_neighbors script
!1840 Fix Eltex.RG.get_version script
!1841 Fix Eltex.DSLAM profile
!1842 Fix Hiawei.MA5300. Disable session use.
!1844 Fix Qtech.QSW2500 script
!1852 Fix ManagedObject card not shows interfaces without metrics.
!1857 #1014 Fix ‘Config MUST NOT match string’ validator
!1861 Fix row_limit in web app.
!1864 Fix ch-policy dot name on command.
!1867 Fix ManagedObject vendor migration.
!1869 Fix wrong scripts name
!1871 Fix tests
!1874 py3: Fix threadpool
!1875 fix get_config tplink t2600 confdb
!1876 Fix tests
!1889 Fix MikroTik.RouterOS profile
!1890 Fix SKS.SKS.get_spanning_tree script
!1897 fix get_inv for X650
!1899 Fix to_python method for Clickhouse nested field.
!1901 Fix SKS.SKS convert_interface_name() function
!1902 Fix SKS.SKS.get_spanning_tree script
!1903 Fix Iskratel.ESCOM.get_spanning_tree script
!1904 fix_metric_memory
!1905 Fix Nateks.FlexGain.get_interfaces script
!1908 Fix limit check on extapplication.
!1928 Fix DLink.DxS.get_interface_status_ex script
!1929 Revert “Merge branch ‘qsw2800-more-patterns’ into ‘master’”
!1930 Fix Alstec.24xx.get_interfaces script
!1931 Catch SNMP Error when Huawei.VRP SNMP worked.
!1938 Fix classifier rule cleanup code.
!1939 Some fixes for Hikvision.DSKV8 profile.
!1940 Fix get_handler error message typo.
!1942 Fix CPEStatus discovery when calculate diff for new cpe.
!1943 Fix Huawei.MA5600T pattern more for Honet.
!1946 Fix multiple jobs on master branch conflict. Prettify a little bit.
!1950 Fix Huawei.VRP.get_capabilities snmp check when get.
!1954 Some fixes in Generic snmp scripts.
!1961 Fix choices attribute in StringList interface parameter.
!1966 Fix Juniper.JunOS confdb normalizer.
!1967 Fix job_handler validator in Workflow editor backend.
!1979 Fix Cisco.IOS.get_version script
!1986 LineTokenizer: Fix string_quote processing
!1987 Fix Qtech.QSW2800.get_inventory datetime import.
!1989 Fix help command execute on custom.
!1991 Fix get_config script on profiles for policy.
!1993 #1041 Fix segment update_summary fix.
!1994 #1039 Make IPAM VRF search ungrouped VRF.
!1996 Fix calculate address and prefix usage.
!2001 Fix reportdatasource backend.
!2003 #1044 Fix import error on reportoverlappedoids.
!2004 #981 Fix Cisco.ASA pattern_username match to Last login message.

Code cleanup

MR Title
!1591 Remove wildcard imports
!1604 DocInline code cleanup
!1605 Remove inv and fm monitor deprecated API
!1606 Remove cocomo report
!1771 fix_YAMLLoad_safe_load
!1792 Set yapf column limit to 120
!1797 flake8: Disable W605
!1808 cm_migrations. code refactoring
!1809 crm_migrations. code refactoring
!1810 dns_migrations. code refactoring
!1812 fm_migrations. code refactoring
!1813 gis_migrations. code refactoring
!1815 inv_migrations. code refactoring.
!1818 ip_migrations. code refactoring
!1819 kb_migrations. code refactoring
!1821 main_migrations. code refactoring
!1823 peer_migrations. code refactoring
!1824 phone_migrations. code refactoring
!1825 pm_migrations. code refactoring
!1826 project_migrations. code refactoring
!1828 sa_migrations. code refactoring
!1829 sla_migrations. code refactoring
!1830 vc_migrations. code refactoring
!1831 wf_migrations. code refactoring.
!1832 Update init.py
!1850 flake8: Fix tests/
!1851 Remove lib/tt.py
!1856 Remove obsolete periodics
!1870 Cleanup noc.settings import.
!1880 py3: Rename izip_longest
!1882 Cleanup init.py
!1891 flake8_E2_E3_ver2 without sa_profiles, services, tests
!1892 fix_migrations. DeprecationWarning pymongo
!1898 flake8_E2_E3_ver2. services
!1909 flake8_E2_E3_ver2. sa_profiles
!1915 Fix @on_delete_check decorators
!1918 Code cleanup
!1958 Cleanup DeprecationWarnings
!1962 Remove socketKeepAlive option
!1968 flake8_f401_webapp_cm
!1969 flake8_f401_webapp_crm
!1970 flake8_f401_webapp_dns
!1971 flake8_f401_webapp_fm
!1972 flake8_f401_webapp_gis
!1973 flake8_f401_webapp_inv
!1974 flake8_f401_webapp_main
!1975 flake8_f401_webapp_maintenance
!1976 lib.app cleanup
!1980 2to3 idioms
!1981 del_with_statement
!1982 2to3 basestring
!1983 2to3 set literal
!1984 Port legacy tests
!1985 2to3 metaclass