In accordance to our Release Policy we’re proudly present release 20.1.

20.1 release contains of 254 bugfixes, optimisations and improvements. Refer to the Release Notes for details.

Highlights

Python3 Compatibility

As it was declared in 19.4 Release Notes, 20.1 is the first release which offers full py3 compatibility. Python version can be selected in the Tower interface during deploy.

Following Python versions are used:

• 2.7 for py2 installations
• 3.6 for py3 installations

You should stay on py2 if:

• CLIPS-based config config validation is used. You’re urged to move to the ConfDB based validation engine.
• custom is used. You have to check custom code base for py3 compatibility with tools like 2to3 and futurize. It is advised to check results on separate test installation.
• pyrules are used. Just in case with custom you have to perform additional compatibility checking and testing is required.
• You have very personal reason to stay on py2.

All other installation should be deployed in py3 mode.

Please note that Python 3.7+ compatibility cannot be reached just now due to dependencies restriction. Reaching latest stable python version compatibility is still work in progress.

Returning to 19.4 Release Notes - we’re starting to remove py2 compatibility code and testing infrastructure in order to clean our codebase. NOC 20.2 will be py3-only release.

Rust infrastructure

NOC beginning to adopt Rust as a viable component of its infrastructure. Rust is blazingly fast and memory-effective language for building reliable software. Common motivation behind the Rust:

• No language-integrated GC, so it integrates well with Python via PyO3 bindings. Rust can fully replace Cython in the NOC project. Following parts of system can gain great benefit from Rust:

  • ASN.1 BER decoder (SNMP Operation)
  • IP Address manipulation
  • HTTP Client

• Can be used to build high-performance existing part of the system:

  • ping
  • chwriter
  • syslogcollector
  • trapcollector

• Can be used to build like new perspective components like:

  • flowcolletor
  • cdrcollector

• Can be used to write standalone components, like server agents

Though some part of Rust usage held by py3 transition (pyo3 supports Python 3.5+), we’re expecting to remove all obstacles in 20.2 release.

We’re already rewritten DNS Synchronization component from Go language to Rust.

Physical to Logical Interface Mapping

New mechanism called Collator has been introduced. Collator establishes connection between logical interface (ManagedObject Interface) and hardware slot bound to inventory model. So it is possible to answer the questions:

• Where particular interface is physically located?
• Has the particular interface any hardware restrictions?
• Which logical port corresponds to the physical slot?
• Which logical ports are served by given hardware module?

Composite Connections

Inventory support for composite connections has been added. Composite connections are physical jacks serving several ports at once. Most known cases are RJ-21/mRJ-21, CENC-36M. Composites are just bundle of pins assigned to several internal connections.

NVD CPE

Inventory support for NVD CPE identifiers has been added. NOC supports CPE v2.2 and v2.3 inventory attributes. NVD CPE identifiers allow to use NOC in security audit process.

Serial Number Checking

Additional inventory model attributes for serial number validity checking has been introduced. Minimal and maximal length of serial number and regular expression pattern can be set.

Binary, broken and non-unique serial numbers are headache on cheap optical modules, so it is good idea to drop obviously crap serial and fallback to NOC-generated ones.

Compare Configs

Configs from different managed objects can be compared via UI

./noc inventory find-serial command

Handy find-serial cli command has been added. It is possible to search for one or more serials from common line and see, where this module is located.

SNMP Display-Hints

SNMP Display-Hints are the first-class citizens. So NOC tries to decode binary OctetString data to the readable textual representation, if defined by MIB or set manually during the function call.

Task Monitor

UI to control discovery jobs has been added.

FM Pools

Now it is possible to separate Managed Object’s SA and FM pool binding. Possible cases are:

• Intersecting address space within single segment
• Eliminating the need to allocate separate classifier/correlator processes for small SA pools.

NBI getmappings API

NBI API to query object mappings has been introduced. Mappings are relations between identifiers in NOC and in the remote systems. Consider NOC loads Managed Object from external network inventory system (NRI) via ETL process. NRI has own ID for the object (remote id), while NOC assigns its own (local id). So mapping is the relation between NOC’s ID, and the pairs of (Remote System, Remote ID).

getmappings API allows to query objects by local or remote id and to get all known mappings in the single call.

Protected Fields

Some fields in UI can be marked as protected, preventing manual user changes.

Development Process Changes

Towards Monorepo

We’re beginning to collect all NOC-related repositories to a single repo. Following repos have been merged:

• collections
• noc-sync-bind

Flake8 Checks

Codebase is clean enough to enforce previously disabled flake8 check, so they are mandatory now:

• F403 - ‘from xxx import *’ used; unable to detect undefined names
• F405 - ‘xxx’ may be undefined, or defined from star imports: xxx
• W605 - invalid escape sequence ‘x’
  

Pending Deprecations

CLIPS-based configuration validation is not supported on python3 installations and will be removed in NOC 20.2. Please select Python 2 if you have config validation rules and consider to rewrite them to ConfDB validation queries or your installation may be stuck on NOC 20.1 release.

New features

MR	Title
!811	Add Task Monitor
!2753	Add Open/Close Events from Thresholds
!2952	noc/noc#1150 Add protected fields to web application backend.
!3063	Lazy MIB loading
!3082	noc/noc#1221 snmp display-hint processing
!3090	noc/noc#1110 Add composite connection.
!3103	noc/noc#1111 Add combo field to objectmodel for Combined connection type.
!3115	Add single ddash for interfaces in card
!3179	Rust infrastructure
!3214	#1219 Additional topology information in managedobject datastream
!3225	nbi: getmappings API
!3235	Interface type ‘internal’
!3297	#1264 confdb: meta matchers section
!3307	#1269: Physical port to logical interface binding
!3311	installation_id and globally-unique bi_id generation
!3312	#1255 FM Pools
!3324	#1277 asset: Serial numbers checking
!3335	./noc inventory find-serial command

Improvements

MR	Title
!2991	Fix Cisco Nexus version detection
!3006	Fix Eltex.LTE.get_inventory script
!3040	managedobject datastream: admin_status field
!3067	mib: Resolve syntax references
!3070	Speedup find service on etl load.
!3072	./noc sync-mib speedup
!3073	Add extra_vars params to pm dash
!3074	./noc compile-mib is moved into ./noc mib make-collection
!3075	Allow to import MIBs in BIG5 encoding
!3088	inv.objectmodel remove renderer on protocols column
!3099	Add query param to confdb command.
!3100	Rebuild datastream when object or objectconnection change.
!3110	Add make collection from directory to mib command. Fix decode error on compile mib API.
!3122	Add iface threshold profile handler
!3139	Interface threshold handling fixes
!3148	noc/noc#1085 Confdb add VRRP. initial version. Add cisco, huawei
!3158	Add queue params to close_tt and add_comment method on TT adapter.
!3159	noc/noc#1184 IGetAlarms SA interface.
!3166	Fix ldap auth for ldap3 verison 2.0.9-2.6.1 (Py3)
!3168	Add promote_affected_tt param to AlarmEscalation config.
!3169	noc/noc#1245 Move configvalidation umbrella to ‘Config
!3171	Add escalate umbrella alarm.
!3187	Update Search by MAC in MO Inteface
!3202	#1203 Span.set_error, .set_error_from_exc helpers
!3206	Replace get_handler on classifier service.
!3215	./noc datastream rebuild –jobs
!3217	Add docs for optical model interface
!3220	Add docs for cpe part of asset model interface
!3232	#1263 bi managedobject: full link methods detalization
!3234	test snmp_metrics json
!3236	noc/noc#1087 ConfDB. add meta chassis_id
!3238	JUnit XML test report
!3242	check-labels: JUnit XML report
!3244	ConfDB: vrrp. add preempt
!3248	pymongo 3.10.1
!3253	rust project label
!3254	mongoengine 0.19.1
!3255	noc/noc#1078 datastream: config section of managedobject datastream
!3262	flake8 JUnit XML report
!3279	Reworked collection JSON sharing
!3293	inv.objectmodels collection tests
!3295	Add ‘Config
!3304	box: move asset discovery just after config one
!3305	Fix service mailsender
!3309	test collections’ uuids for uniqueness
!3313	check-labels: Fix migration label detection
!3314	Retry check-labels as the labels will be fixed from webhook
!3320	Activate collections tests
!3322	add new report - load metric max
!3323	Fix report load metrics
!3328	check-labels: Detect collections label
!3329	DCS: Stop one-time resolvers
!3330	#1191 scheduler: Gracefully apply pending bulk operations on shutdown
!3333	managedobject datastream: Interface services
!3339	Catch IOError exception on beef load.
!3353	Raise error when OID tree from beef end.
!3354	Add display_hints options to get_table script method.
!3355	Restore ujson-x 1.37
!3366	#1278 sae: Pass attributes section

Bugfixes

MR	Title
!2879	Fix Exception when send message ‘ServiceStub’ object has no attribute ‘pub’
!3013	#1186 py3: Fix request.body processing
!3014	#1185 py3: Fix error reporting on missed custom modules
!3015	Fix cleanup 0192_thresholdprofile migration.
!3019	Fix 0153_managedobjectselector_profile migration after fix 0150_managed_object_profile migration.
!3024	noc/noc#1195 Catch ValueError when convert_interface_name on TopologyDiscovery.
!3024	noc/noc#1195 Catch ValueError when convert_interface_name on TopologyDiscovery.
!3029	py3: alnum_key() for reliable sorting
!3034	Fix TypeError on http client when working over proxy.
!3037	#1193 py3: Fix MRT service
!3041	#1201 py3: Fix http client eof_mark handling
!3045	#1196 py3: Fix location handling
!3047	#1209 Fix syslogcollector/trapcollector port binding
!3055	Fix CH objectprofile datasource bool field.
!3062	noc/noc#1197 Fix encoding on interfacepath card.
!3064	py3 fix beef import encoding.
!3066	Add getting Job log to SA Monitor application
!3068	fix Alstec, Eltex pon/rg, qsw8200, dlink get_inventory
!3069	py3: Fix sync-mibs
!3078	Fix typo in text parse_table on row_wrapper param check.
!3093	fix ReportAlarmDetail
!3094	noc/noc#1188 Fix memory leak on SSH CLI.
!3097	noc/noc#1096 Fix correlator trigger handler call.
!3104	noc/noc#1213 py3. Fix TypeError when collected config convert to binary.
!3137	Fix MAC AutoAPI doc builder
!3143	Fix object datastream iter_changed managed_object check.
!3152	Fix object iter_changed_datastream typo.
!3161	Fix Segment MAC Discovery when the object links to itself
!3162	Fix interface_flap field on ReportMetrics.
!3178	Fix import future print in deploy scripts
!3184	Fix promote_affected_tt param when escalation item.
!3201	sa.managedobject: Remove duplicated LAG Profile column
!3205	Fix delay escalation when start time < now time
!3207	#1227: Filter out duplicated metric measurements
!3221	Rebuild credential cache when change ManagedObjectProfile beef_storage and beef_path_template field.
!3224	py3: Fix nbi service logging
!3230	Fix typo in etl extractor.
!3239	Fix Test notification group, when id > 9.
!3247	Add ldap3 escape filter chars
!3250	#1267 Fix Peer.save()
!3258	py3: Fix whois client
!3259	#1210: Better json content type handling
!3263	Add MO queue for maintenance escalation
!3267	Fix Alarms class filter
!3288	Fix VRF type name to lowercase (were changed on interface).
!3291	Add raise_privileges and access_preference params when call login script on suggestcli discovery.
!3303	inv.objectmodels: Fix duplicated uuids
!3327	Fix typo max_serial_size on asset discovery.
!3356	py3: Fix cli.object_stream
!3358	py3: Fix name encoding when calculate config diff.
!3359	Convert cmd_next and cmd_stop to bytes on object_cli parser.
!3371	#1280 py3 ConfDB: Fix node sorting
!3373	py3: Fix adaptive prompt pattern
!3374	py3: Fix telnet sb/se logging
!3376	py3: Clean ETL loader stream from unreadable characters
!3379	Add fm_pool to field managed object loader.
!3381	py3: Fix Selectors over attributes
!3384	Fix NRI discovery

Code Cleanup

MR	Title
!3036	noc/noc#1198 Fix AlarmDiagnostic.get_diagnostics encode.
!3053	Update docs/src/en/services/icqsender.rst
!3060	Fix activator.http_get decode error.
!3061	noc/noc#1197 Fix sortdict on managedobject card.
!3071	py3. Fix MIBAPI service.
!3102	LLDP-MIB render test
!3107	Update requirements/deploy scripts for py3/py2 compat
!3138	Links to MR list
!3195	py3. Not encoding utf-8 on geocodercache.
!3200	noc/noc#1250 Fix ETL for py3.
!3204	noc/noc#1256 Fix future.set_exception_info on threadpool for py3.
!3208	Remove 20.1 deprecations
!3209	Bump pynsq version to 0.8.3
!3216	Code formatting
!3219	Fix bytes on script super_username submit.
!3233	py3: Fix bi extract
!3249	flake8: Enable W605 check
!3251	rust: remove “extern crate”
!3252	CI: Remove migrate stages
!3260	Remove deprecated models and applications
!3261	RCA neighbor cache is default and only
!3264	ujson 2.0.2
!3285	Merge back collections/ to main repo
!3286	Restore tests
!3296	flake8: Enable F403 and F405 checks
!3310	inv.connectiontypes UUID uniqueueness test
!3315	Fix cr_context on model
!3316	Replace gender for c14 connection type part1
!3317	Replace gender for c14 connection type part2
!3318	Replace gender for c14 connection type part3
!3319	Replace gender for c14 connection type part4
!3336	py3: mailsender fixes
!3338	Fix typo on asset discovery logging.
!3340	Replace StringIO to BytesIO on DetailReport for xlsxwriter work.
!3350	py3. Fix expected byte trace on interface-profile command.
!3352	py3: Convert rogue_chars to binary type
!3357	py3: Fix if between None and int type.
!3362	py3: Disable CLIPS parsers.
!3365	Clean data type on Consul config client.
!3372	py3: Replace split_alnum to alnum_key on matchers.
!3375	py3: Replace decode method to codecs on profiles.
!3377	py3: Convert addr and port to bytes on http client.
!3378	py3: Convert hash on geocodercache to text.

Profile Changes

Alcatel.7302

MR	Title
!3089	Alcatel 7302 fix port_map
!3123	fix Alcatel 7302 serial
!3243	Alcatel.7302 fix empty b_serial, b_revision. fix port_map. version 2

Alsitec.24xx

MR	Title
!3119	noc/noc#1181 Alstec.24xx. Fix unknown serial number, and devices with 4413 DeviceID.
!3128	noc/noc#1181 Alstec.24xx.get_inventory. Fallback to CLI when unknown SNMP part_no.
!3144	Alstec.24xx.get_inventory. Set script prefer to S.
!3144	Alstec.24xx.get_inventory. Set script prefer to S.

Cisco.IOS

MR	Title
!3009	Update Cisco.IOS profile to support old hardware
!3173	Cisco.IOS.get_lacp_neighbors, get_version. Small fixes encoding.
!3185	Cisco.IOS.get_lacp_neighbors. Fix rx_sys_id regex.
!3229	Cisco.IOS.get_version and get_capabilities cleanup code.
!3229	Cisco.IOS.get_version and get_capabilities cleanup code.
!3237	Cisco.IOS.get_switchport. Fix getting vlans from SNMP for python3.
!3342	Cisco.IOS.get_cdp_neighbors. Replace OID to mib name.
!3351	Cisco.IOS.get_inventory. Use matchers instead match.

DLink.DxS

MR	Title
!3091	Bypass confirmation of firmware upgrade on DGS-1100-10/ME
!3363	DLink.DxS. Fix decode on profile.
!3367	Dlink.DxS.get_interfaces. Remove decode ascii interface description.

Dahua.DH

MR	Title
!3142	Dahua.DH. Fix getting config from RVi web cam.
!3174	noc/noc#1216 Dahua.DH.get_version. Fix script on cam without PTZ module.

Eltex.MES

MR	Title
!3155	add_metrics_SensorStatus_Eltex_MES
!3177	fix_Environment_Sensor_Status_Eltex_MES

Eltex.MES24xx

MR	Title
!3007	Fix Eltex.MES24xx.get_lldp_neighbors script
!3164	fix_get_lldp_neighbors_Eltex_MES24xx

Generic

MR	Title
!3020	Generic.get_lldp_neighbors. Strip \x00 symbol on SNMP string.
!3023	Generic.get_capabilities. Use cache value sysObjectID for getting EnterpriseID.
!3023	Generic.get_capabilities. Use cache value sysObjectID for getting EnterpriseID.
!3032	Generic.get_beef. Fix fsm collecting - not collected if get_version first.
!3084	Generic.get_capabilities. Fix bad enterprise id check.
!3149	Generic.get_beef. Catch SNMP error when collected beef.
!3218	py3. Message attribute not supported on Exception.
!3348	Generic.get_switchport. Fix getting vlans on SNMP.

H3C.VRP

MR	Title
!3240	H3C.VRP. Fix get_version and add SNMP scripts.
!3300	H3C.VRP.get_version. Fix regex for S3100-8TP-EI model.

Huawei.MA5600T

MR	Title
!3022	Huawei.MA5600T. Fix lstrip on profile parse_table.
!3163	noc/noc#1226 Huawei.MA5600T. Add pattern_more for MA5800-X2 platform.
!3163	noc/noc#1226 Huawei.MA5600T. Add pattern_more for MA5800-X2 platform.
!3337	Huawei. Disable CLI metrics.

Huawei.VRP

MR	Title
!3105	Huawei.VRP. Add S3328 to hw_series.
!3113	Huawei.VRP. Fix version and inventory script for CX300 platform.
!3116	noc/noc#1163 Huawei.VRP.get_inventory. Add S8504 slot count.
!3126	noc/noc#1163 Huawei.VRP. Add error_pattern_prompt and fix get_version encoding.
!3345	Huawei.VRP.get_switchport. Fix getting port vlan on SNMP.

Iskratel.MSAN

MR	Title
!3344	Fix send_on_syntax_error params for py3.

Juniper.JUNOS

MR	Title
!3058	Juniper.JUNOS.normalizer. Fix detect vrf type.
!3256	add snmp oid jnxVirtualChassisMemberMacAddBase. fix get_chassis_id Juniper_JUNOS

Qtech.QSW2800

MR	Title
!2960	Add support for stack in QSW-8200-28F-AC-DC
!3222	Qtech.QSW2800.get_capabilities. Fix not supported operation on ‘NoneType’ and ‘int’ type.
!3343	Qtech.QSW2800.get_fqdn. Fix py3 pattern encoding.

SKS.SKS

MR	Title
!3127	noc/noc#1180 SKS.SKS. Fix get_spanning_tree designated_port_id and get_inventory unknown E1 module.

Siklu.EH

MR	Title
!3364	Siklu.EH. Fix cleaned_input to binary data.

rare

MR	Title
!3038	Add NSN.hiX56xx.get_inventory script
!3050	DCN.DCWL. Fix profile for SNMP
!3052	DCN.DCWL.get_version. Fix typo on matchers.
!3079	Carelink.SWG. Fix pattern_more on profile.
!3083	Qtech.QSW8200.get_lldp_neighbors. Add ifName to port_type.
!3108	fix_get_inventory_Eltex_ESR_profile
!3150	fix_AlliedTelesis.AT9900_profile
!3160	add_get_ifindexes_AlliedTelesis.AT9900_profile
!3167	add_get_lldp_neighbors_AlliedTelesis.AT9900
!3180	add_cli_get_lldp_neighbors_AlliedTelesis.AT9900
!3186	add_get_capabilities_AlliedTelesis.AT9900
!3196	Fix get_interfaces for BDCOM.xPON profile
!3241	Fix invalid JSON
!3245	Fix get_interfaces NSN.TIMOS
!3298	fix_get_interfaces - port-channel no has iftype
!3306	NSN.TIMOS Fix get_intefaces script
!3308	Iscratel.ESCOM fix profile
!3332	Iskratel.ESCOM fix lldp neigborg script
!3346	py3. Cleanup get_lldp_neighbors script for SNMP.
!3360	fix_get_config_NSN.hiX56xx
!3361	fix_pattern_more_Huawei_VRP3
!3369	Fix Check metrics collect on DCN and Rotek profiles.
!3370	py3: Convert telnet_send_on_connect to binary.
!3382	NSN.TIMOS.get_interfaces. Fix empty mac address on tunnel interface.