NOC 20.1


In accordance to our Release Policy we’re proudly present release 20.1.

20.1 release contains of 254 bugfixes, optimisations and improvements. Refer to the Release Notes for details.


Python3 Compatibility

As it was declared in 19.4 Release Notes, 20.1 is the first release which offers full py3 compatibility. Python version can be selected in the Tower interface during deploy.

Following Python versions are used:

  • 2.7 for py2 installations
  • 3.6 for py3 installations

You should stay on py2 if:

  • CLIPS-based config config validation is used. You’re urged to move to the ConfDB based validation engine.
  • custom is used. You have to check custom code base for py3 compatibility with tools like 2to3 and futurize. It is advised to check results on separate test installation.
  • pyrules are used. Just in case with custom you have to perform additional compatibility checking and testing is required.
  • You have very personal reason to stay on py2.

All other installation should be deployed in py3 mode.

Please note that Python 3.7+ compatibility cannot be reached just now due to dependencies restriction. Reaching latest stable python version compatibility is still work in progress.

Returning to 19.4 Release Notes - we’re starting to remove py2 compatibility code and testing infrastructure in order to clean our codebase. NOC 20.2 will be py3-only release.

Rust infrastructure

NOC beginning to adopt Rust as a viable component of its infrastructure. Rust is blazingly fast and memory-effective language for building reliable software. Common motivation behind the Rust:

  • No language-integrated GC, so it integrates well with Python via PyO3 bindings. Rust can fully replace Cython in the NOC project. Following parts of system can gain great benefit from Rust:

    • ASN.1 BER decoder (SNMP Operation)
    • IP Address manipulation
    • HTTP Client
  • Can be used to build high-performance existing part of the system:

    • ping
    • chwriter
    • syslogcollector
    • trapcollector
  • Can be used to build like new perspective components like:

    • flowcolletor
    • cdrcollector
  • Can be used to write standalone components, like server agents

Though some part of Rust usage held by py3 transition (pyo3 supports Python 3.5+), we’re expecting to remove all obstacles in 20.2 release.

We’re already rewritten DNS Synchronization component from Go language to Rust.

Physical to Logical Interface Mapping

New mechanism called Collator has been introduced. Collator establishes connection between logical interface (ManagedObject Interface) and hardware slot bound to inventory model. So it is possible to answer the questions:

  • Where particular interface is physically located?
  • Has the particular interface any hardware restrictions?
  • Which logical port corresponds to the physical slot?
  • Which logical ports are served by given hardware module?

Composite Connections

Inventory support for composite connections has been added. Composite connections are physical jacks serving several ports at once. Most known cases are RJ-21/mRJ-21, CENC-36M. Composites are just bundle of pins assigned to several internal connections.


Inventory support for NVD CPE identifiers has been added. NOC supports CPE v2.2 and v2.3 inventory attributes. NVD CPE identifiers allow to use NOC in security audit process.

Serial Number Checking

Additional inventory model attributes for serial number validity checking has been introduced. Minimal and maximal length of serial number and regular expression pattern can be set.

Binary, broken and non-unique serial numbers are headache on cheap optical modules, so it is good idea to drop obviously crap serial and fallback to NOC-generated ones.

Compare Configs

Configs from different managed objects can be compared via UI

./noc inventory find-serial command

Handy find-serial cli command has been added. It is possible to search for one or more serials from common line and see, where this module is located.

SNMP Display-Hints

SNMP Display-Hints are the first-class citizens. So NOC tries to decode binary OctetString data to the readable textual representation, if defined by MIB or set manually during the function call.

Task Monitor

UI to control discovery jobs has been added.

FM Pools

Now it is possible to separate Managed Object’s SA and FM pool binding. Possible cases are:

  • Intersecting address space within single segment
  • Eliminating the need to allocate separate classifier/correlator processes for small SA pools.

NBI getmappings API

NBI API to query object mappings has been introduced. Mappings are relations between identifiers in NOC and in the remote systems. Consider NOC loads Managed Object from external network inventory system (NRI) via ETL process. NRI has own ID for the object (remote id), while NOC assigns its own (local id). So mapping is the relation between NOC’s ID, and the pairs of (Remote System, Remote ID).

getmappings API allows to query objects by local or remote id and to get all known mappings in the single call.

Protected Fields

Some fields in UI can be marked as protected, preventing manual user changes.

Development Process Changes

Towards Monorepo

We’re beginning to collect all NOC-related repositories to a single repo. Following repos have been merged:

  • collections
  • noc-sync-bind

Flake8 Checks

Codebase is clean enough to enforce previously disabled flake8 check, so they are mandatory now:

  • F403 - ‘from xxx import *’ used; unable to detect undefined names
  • F405 - ‘xxx’ may be undefined, or defined from star imports: xxx
  • W605 - invalid escape sequence ‘x’

Pending Deprecations

CLIPS-based configuration validation is not supported on python3 installations and will be removed in NOC 20.2. Please select Python 2 if you have config validation rules and consider to rewrite them to ConfDB validation queries or your installation may be stuck on NOC 20.1 release.

New features

MR Title
!811 Add Task Monitor
!2753 Add Open/Close Events from Thresholds
!2952 noc/noc#1150 Add protected fields to web application backend.
!3063 Lazy MIB loading
!3082 noc/noc#1221 snmp display-hint processing
!3090 noc/noc#1110 Add composite connection.
!3103 noc/noc#1111 Add combo field to objectmodel for Combined connection type.
!3115 Add single ddash for interfaces in card
!3179 Rust infrastructure
!3214 #1219 Additional topology information in managedobject datastream
!3225 nbi: getmappings API
!3235 Interface type ‘internal’
!3297 #1264 confdb: meta matchers section
!3307 #1269: Physical port to logical interface binding
!3311 installation_id and globally-unique bi_id generation
!3312 #1255 FM Pools
!3324 #1277 asset: Serial numbers checking
!3335 ./noc inventory find-serial command


MR Title
!2991 Fix Cisco Nexus version detection
!3006 Fix Eltex.LTE.get_inventory script
!3040 managedobject datastream: admin_status field
!3067 mib: Resolve syntax references
!3070 Speedup find service on etl load.
!3072 ./noc sync-mib speedup
!3073 Add extra_vars params to pm dash
!3074 ./noc compile-mib is moved into ./noc mib make-collection
!3075 Allow to import MIBs in BIG5 encoding
!3088 inv.objectmodel remove renderer on protocols column
!3099 Add query param to confdb command.
!3100 Rebuild datastream when object or objectconnection change.
!3110 Add make collection from directory to mib command. Fix decode error on compile mib API.
!3122 Add iface threshold profile handler
!3139 Interface threshold handling fixes
!3148 noc/noc#1085 Confdb add VRRP. initial version. Add cisco, huawei
!3158 Add queue params to close_tt and add_comment method on TT adapter.
!3159 noc/noc#1184 IGetAlarms SA interface.
!3166 Fix ldap auth for ldap3 verison 2.0.9-2.6.1 (Py3)
!3168 Add promote_affected_tt param to AlarmEscalation config.
!3169 noc/noc#1245 Move configvalidation umbrella to ‘Config
!3171 Add escalate umbrella alarm.
!3187 Update Search by MAC in MO Inteface
!3202 #1203 Span.set_error, .set_error_from_exc helpers
!3206 Replace get_handler on classifier service.
!3215 ./noc datastream rebuild –jobs
!3217 Add docs for optical model interface
!3220 Add docs for cpe part of asset model interface
!3232 #1263 bi managedobject: full link methods detalization
!3234 test snmp_metrics json
!3236 noc/noc#1087 ConfDB. add meta chassis_id
!3238 JUnit XML test report
!3242 check-labels: JUnit XML report
!3244 ConfDB: vrrp. add preempt
!3248 pymongo 3.10.1
!3253 rust project label
!3254 mongoengine 0.19.1
!3255 noc/noc#1078 datastream: config section of managedobject datastream
!3262 flake8 JUnit XML report
!3279 Reworked collection JSON sharing
!3293 inv.objectmodels collection tests
!3295 Add ‘Config
!3304 box: move asset discovery just after config one
!3305 Fix service mailsender
!3309 test collections’ uuids for uniqueness
!3313 check-labels: Fix migration label detection
!3314 Retry check-labels as the labels will be fixed from webhook
!3320 Activate collections tests
!3322 add new report - load metric max
!3323 Fix report load metrics
!3328 check-labels: Detect collections label
!3329 DCS: Stop one-time resolvers
!3330 #1191 scheduler: Gracefully apply pending bulk operations on shutdown
!3333 managedobject datastream: Interface services
!3339 Catch IOError exception on beef load.
!3353 Raise error when OID tree from beef end.
!3354 Add display_hints options to get_table script method.
!3355 Restore ujson-x 1.37
!3366 #1278 sae: Pass attributes section


MR Title
!2879 Fix Exception when send message ‘ServiceStub’ object has no attribute ‘pub’
!3013 #1186 py3: Fix request.body processing
!3014 #1185 py3: Fix error reporting on missed custom modules
!3015 Fix cleanup 0192_thresholdprofile migration.
!3019 Fix 0153_managedobjectselector_profile migration after fix 0150_managed_object_profile migration.
!3024 noc/noc#1195 Catch ValueError when convert_interface_name on TopologyDiscovery.
!3024 noc/noc#1195 Catch ValueError when convert_interface_name on TopologyDiscovery.
!3029 py3: alnum_key() for reliable sorting
!3034 Fix TypeError on http client when working over proxy.
!3037 #1193 py3: Fix MRT service
!3041 #1201 py3: Fix http client eof_mark handling
!3045 #1196 py3: Fix location handling
!3047 #1209 Fix syslogcollector/trapcollector port binding
!3055 Fix CH objectprofile datasource bool field.
!3062 noc/noc#1197 Fix encoding on interfacepath card.
!3064 py3 fix beef import encoding.
!3066 Add getting Job log to SA Monitor application
!3068 fix Alstec, Eltex pon/rg, qsw8200, dlink get_inventory
!3069 py3: Fix sync-mibs
!3078 Fix typo in text parse_table on row_wrapper param check.
!3093 fix ReportAlarmDetail
!3094 noc/noc#1188 Fix memory leak on SSH CLI.
!3097 noc/noc#1096 Fix correlator trigger handler call.
!3104 noc/noc#1213 py3. Fix TypeError when collected config convert to binary.
!3137 Fix MAC AutoAPI doc builder
!3143 Fix object datastream iter_changed managed_object check.
!3152 Fix object iter_changed_datastream typo.
!3161 Fix Segment MAC Discovery when the object links to itself
!3162 Fix interface_flap field on ReportMetrics.
!3178 Fix import future print in deploy scripts
!3184 Fix promote_affected_tt param when escalation item.
!3201 sa.managedobject: Remove duplicated LAG Profile column
!3205 Fix delay escalation when start time < now time
!3207 #1227: Filter out duplicated metric measurements
!3221 Rebuild credential cache when change ManagedObjectProfile beef_storage and beef_path_template field.
!3224 py3: Fix nbi service logging
!3230 Fix typo in etl extractor.
!3239 Fix Test notification group, when id > 9.
!3247 Add ldap3 escape filter chars
!3250 #1267 Fix
!3258 py3: Fix whois client
!3259 #1210: Better json content type handling
!3263 Add MO queue for maintenance escalation
!3267 Fix Alarms class filter
!3288 Fix VRF type name to lowercase (were changed on interface).
!3291 Add raise_privileges and access_preference params when call login script on suggestcli discovery.
!3303 inv.objectmodels: Fix duplicated uuids
!3327 Fix typo max_serial_size on asset discovery.
!3356 py3: Fix cli.object_stream
!3358 py3: Fix name encoding when calculate config diff.
!3359 Convert cmd_next and cmd_stop to bytes on object_cli parser.
!3371 #1280 py3 ConfDB: Fix node sorting
!3373 py3: Fix adaptive prompt pattern
!3374 py3: Fix telnet sb/se logging
!3376 py3: Clean ETL loader stream from unreadable characters
!3379 Add fm_pool to field managed object loader.
!3381 py3: Fix Selectors over attributes
!3384 Fix NRI discovery

Code Cleanup

MR Title
!3036 noc/noc#1198 Fix AlarmDiagnostic.get_diagnostics encode.
!3053 Update docs/src/en/services/icqsender.rst
!3060 Fix activator.http_get decode error.
!3061 noc/noc#1197 Fix sortdict on managedobject card.
!3071 py3. Fix MIBAPI service.
!3102 LLDP-MIB render test
!3107 Update requirements/deploy scripts for py3/py2 compat
!3138 Links to MR list
!3195 py3. Not encoding utf-8 on geocodercache.
!3200 noc/noc#1250 Fix ETL for py3.
!3204 noc/noc#1256 Fix future.set_exception_info on threadpool for py3.
!3208 Remove 20.1 deprecations
!3209 Bump pynsq version to 0.8.3
!3216 Code formatting
!3219 Fix bytes on script super_username submit.
!3233 py3: Fix bi extract
!3249 flake8: Enable W605 check
!3251 rust: remove “extern crate”
!3252 CI: Remove migrate stages
!3260 Remove deprecated models and applications
!3261 RCA neighbor cache is default and only
!3264 ujson 2.0.2
!3285 Merge back collections/ to main repo
!3286 Restore tests
!3296 flake8: Enable F403 and F405 checks
!3310 inv.connectiontypes UUID uniqueueness test
!3315 Fix cr_context on model
!3316 Replace gender for c14 connection type part1
!3317 Replace gender for c14 connection type part2
!3318 Replace gender for c14 connection type part3
!3319 Replace gender for c14 connection type part4
!3336 py3: mailsender fixes
!3338 Fix typo on asset discovery logging.
!3340 Replace StringIO to BytesIO on DetailReport for xlsxwriter work.
!3350 py3. Fix expected byte trace on interface-profile command.
!3352 py3: Convert rogue_chars to binary type
!3357 py3: Fix if between None and int type.
!3362 py3: Disable CLIPS parsers.
!3365 Clean data type on Consul config client.
!3372 py3: Replace split_alnum to alnum_key on matchers.
!3375 py3: Replace decode method to codecs on profiles.
!3377 py3: Convert addr and port to bytes on http client.
!3378 py3: Convert hash on geocodercache to text.

Profile Changes


MR Title
!3089 Alcatel 7302 fix port_map
!3123 fix Alcatel 7302 serial
!3243 Alcatel.7302 fix empty b_serial, b_revision. fix port_map. version 2


MR Title
!3119 noc/noc#1181 Alstec.24xx. Fix unknown serial number, and devices with 4413 DeviceID.
!3128 noc/noc#1181 Alstec.24xx.get_inventory. Fallback to CLI when unknown SNMP part_no.
!3144 Alstec.24xx.get_inventory. Set script prefer to S.
!3144 Alstec.24xx.get_inventory. Set script prefer to S.


MR Title
!3009 Update Cisco.IOS profile to support old hardware
!3173 Cisco.IOS.get_lacp_neighbors, get_version. Small fixes encoding.
!3185 Cisco.IOS.get_lacp_neighbors. Fix rx_sys_id regex.
!3229 Cisco.IOS.get_version and get_capabilities cleanup code.
!3229 Cisco.IOS.get_version and get_capabilities cleanup code.
!3237 Cisco.IOS.get_switchport. Fix getting vlans from SNMP for python3.
!3342 Cisco.IOS.get_cdp_neighbors. Replace OID to mib name.
!3351 Cisco.IOS.get_inventory. Use matchers instead match.
MR Title
!3091 Bypass confirmation of firmware upgrade on DGS-1100-10/ME
!3363 DLink.DxS. Fix decode on profile.
!3367 Dlink.DxS.get_interfaces. Remove decode ascii interface description.


MR Title
!3142 Dahua.DH. Fix getting config from RVi web cam.
!3174 noc/noc#1216 Dahua.DH.get_version. Fix script on cam without PTZ module.


MR Title
!3155 add_metrics_SensorStatus_Eltex_MES
!3177 fix_Environment_Sensor_Status_Eltex_MES


MR Title
!3007 Fix Eltex.MES24xx.get_lldp_neighbors script
!3164 fix_get_lldp_neighbors_Eltex_MES24xx


MR Title
!3020 Generic.get_lldp_neighbors. Strip \x00 symbol on SNMP string.
!3023 Generic.get_capabilities. Use cache value sysObjectID for getting EnterpriseID.
!3023 Generic.get_capabilities. Use cache value sysObjectID for getting EnterpriseID.
!3032 Generic.get_beef. Fix fsm collecting - not collected if get_version first.
!3084 Generic.get_capabilities. Fix bad enterprise id check.
!3149 Generic.get_beef. Catch SNMP error when collected beef.
!3218 py3. Message attribute not supported on Exception.
!3348 Generic.get_switchport. Fix getting vlans on SNMP.


MR Title
!3240 H3C.VRP. Fix get_version and add SNMP scripts.
!3300 H3C.VRP.get_version. Fix regex for S3100-8TP-EI model.


MR Title
!3022 Huawei.MA5600T. Fix lstrip on profile parse_table.
!3163 noc/noc#1226 Huawei.MA5600T. Add pattern_more for MA5800-X2 platform.
!3163 noc/noc#1226 Huawei.MA5600T. Add pattern_more for MA5800-X2 platform.
!3337 Huawei. Disable CLI metrics.


MR Title
!3105 Huawei.VRP. Add S3328 to hw_series.
!3113 Huawei.VRP. Fix version and inventory script for CX300 platform.
!3116 noc/noc#1163 Huawei.VRP.get_inventory. Add S8504 slot count.
!3126 noc/noc#1163 Huawei.VRP. Add error_pattern_prompt and fix get_version encoding.
!3345 Huawei.VRP.get_switchport. Fix getting port vlan on SNMP.


MR Title
!3344 Fix send_on_syntax_error params for py3.


MR Title
!3058 Juniper.JUNOS.normalizer. Fix detect vrf type.
!3256 add snmp oid jnxVirtualChassisMemberMacAddBase. fix get_chassis_id Juniper_JUNOS


MR Title
!2960 Add support for stack in QSW-8200-28F-AC-DC
!3222 Qtech.QSW2800.get_capabilities. Fix not supported operation on ‘NoneType’ and ‘int’ type.
!3343 Qtech.QSW2800.get_fqdn. Fix py3 pattern encoding.


MR Title
!3127 noc/noc#1180 SKS.SKS. Fix get_spanning_tree designated_port_id and get_inventory unknown E1 module.


MR Title
!3364 Siklu.EH. Fix cleaned_input to binary data.


MR Title
!3038 Add NSN.hiX56xx.get_inventory script
!3050 DCN.DCWL. Fix profile for SNMP
!3052 DCN.DCWL.get_version. Fix typo on matchers.
!3079 Carelink.SWG. Fix pattern_more on profile.
!3083 Qtech.QSW8200.get_lldp_neighbors. Add ifName to port_type.
!3108 fix_get_inventory_Eltex_ESR_profile
!3150 fix_AlliedTelesis.AT9900_profile
!3160 add_get_ifindexes_AlliedTelesis.AT9900_profile
!3167 add_get_lldp_neighbors_AlliedTelesis.AT9900
!3180 add_cli_get_lldp_neighbors_AlliedTelesis.AT9900
!3186 add_get_capabilities_AlliedTelesis.AT9900
!3196 Fix get_interfaces for BDCOM.xPON profile
!3241 Fix invalid JSON
!3245 Fix get_interfaces NSN.TIMOS
!3298 fix_get_interfaces - port-channel no has iftype
!3306 NSN.TIMOS Fix get_intefaces script
!3308 Iscratel.ESCOM fix profile
!3332 Iskratel.ESCOM fix lldp neigborg script
!3346 py3. Cleanup get_lldp_neighbors script for SNMP.
!3360 fix_get_config_NSN.hiX56xx
!3361 fix_pattern_more_Huawei_VRP3
!3369 Fix Check metrics collect on DCN and Rotek profiles.
!3370 py3: Convert telnet_send_on_connect to binary.
!3382 NSN.TIMOS.get_interfaces. Fix empty mac address on tunnel interface.