Skip to content

NOC 20.1

In accordance to our Release Policy <releases-policy> we're proudly present release 20.1.

20.1 release contains 254 bugfixes, optimisations and improvements.

Highlights

Python3 Compatibility

As it was declared in Python3 Compatibility Statement<release-19.4-py3>, 20.1 is the first release which offers full py3 compatibility. Python version can be selected in the Tower interface during deploy.

Following Python versions are used:

  • 2.7 for py2 installations
  • 3.6 for py3 installations

You should stay on py2 if:

  • CLIPS-based config config validation is used. You're urged to move to the ConfDB based validation engine.
  • custom is used. You have to check custom code base for py3 compatibility with tools like 2to3 and futurize. It is advised to check results on separate test installation.
  • pyrules are used. Just in case with custom you have to perform additional compatibility checking and testing is required.
  • You have very personal reason to stay on py2.

All other installation should be deployed in py3 mode.

Please note that Python 3.7+ compatibility cannot be reached just now due to dependencies restriction. Reaching latest stable python version compatibility is still work in progress.

Returning to Python3 Compatibility Statement<release-19.4-py3> -we're starting to remove py2 compatibility code and testing infrastructure in order to clean our codebase. NOC 20.2 will be py3-only release.

Rust infrastructure

NOC beginning to adopt Rust as a viable component of its infrastructure. Rust is blazingly fast and memory-effective language for building reliable software. Common motivation behind the Rust:

  • No language-integrated GC, so it integrates well with Python via PyO3 bindings. Rust can fully replace Cython in the NOC project. Following parts of system can gain great benefit from Rust:
  • ASN.1 BER decoder (SNMP Operation)
  • IP Address manipulation
  • HTTP Client
  • Can be used to build high-performance existing part of the system:
  • ping
  • chwriter
  • syslogcollector
  • trapcollector
  • Can be used to build like new perspective components like:
  • flowcolletor
  • cdrcollector
  • Can be used to write standalone components, like server agents

Though some part of Rust usage held by py3 transition (pyo3 supports Python 3.5+), we're expecting to remove all obstacles in 20.2 release.

We're already rewritten DNS Synchronization component from Go language to Rust.

Physical to Logical Interface Mapping

New mechanism called Collator has been introduced. Collator establishes connection between logical interface (ManagedObject Interface) and hardware slot bound to inventory model. So it is possible to answer the questions:

  • Where particular interface is physically located?
  • Has the particular interface any hardware restrictions?
  • Which logical port corresponds to the physical slot?
  • Which logical ports are served by given hardware module?

Composite Connections

Inventory support for composite connections has been added. Composite connections are physical jacks serving several ports at once. Most known cases are RJ-21/mRJ-21, CENC-36M. Composites are just bundle of pins assigned to several internal connections.

NVD CPE

Inventory support for NVD CPE identifiers has been added. NOC supports CPE v2.2 and v2.3 inventory attributes. NVD CPE identifiers allow to use NOC in security audit process.

Serial Number Checking

Additional inventory model attributes for serial number validity checking has been introduced. Minimal and maximal length of serial number and regular expression pattern can be set.

Binary, broken and non-unique serial numbers are headache on cheap optical modules, so it is good idea to drop obviously crap serial and fallback to NOC-generated ones.

./noc inventory find-serial command

Handy find-serial cli command has been added. It is possible to search for one or more serials from common line and see, where this module is located.

SNMP Display-Hints

SNMP Display-Hints are the first-class citizens. So NOC tries to decode binary OctetString data to the readable textual representation, if defined by MIB or set manually during the function call.

Task Monitor

UI to control discovery jobs has been added.

FM Pools

Now it is possible to separate Managed Object's SA and FM pool binding. Possible cases are:

  • Intersecting address space within single segment
  • Eliminating the need to allocate separate classifier/correlator processes for small SA pools.

NBI getmappings API

NBI API to query object mappings has been introduced. Mappings are relations between identifiers in NOC and in the remote systems. Consider NOC loads Managed Object from external network inventory system (NRI) via ETL process. NRI has own ID for the object (remote id), while NOC assigns its own (local id). So mapping is the relation between NOC's ID, and the pairs of (Remote System, Remote ID).

getmappings API allows to query objects by local or remote id and to get all known mappings in the single call.

Protected Fields

Some fields in UI can be marked as protected, preventing manual user changes.

Development Process Changes

Towards Monorepo

We're beginning to collect all NOC-related repositories to a single repo. Following repos have been merged:

  • collections
  • noc-sync-bind

Flake8 Checks

Codebase is clean enough to enforce previously disabled flake8 check, so they are mandatory now:

  • F403 - 'from xxx import *' used; unable to detect undefined names
  • F405 - 'xxx' may be undefined, or defined from star imports: xxx
  • W605 - invalid escape sequence 'x'

Pending Deprecations

CLIPS-based configuration validation is not supported on python3 installations and will be removed in NOC 20.2. Please select Python 2 if you have config validation rules and consider to rewrite them to ConfDB validation queries or your installation may be stuck on NOC 20.1 release.

New features

MRTitle
MR811Add Task Monitor
MR2753Add Open/Close Events from Thresholds
MR2952noc/noc#1150 Add protected fields to web application backend.
MR3063Lazy MIB loading
MR3082noc/noc#1221 snmp display-hint processing
MR3090noc/noc#1110 Add composite connection.
MR3103noc/noc#1111 Add combo field to objectmodel for Combined connection type.
MR3115Add single ddash for interfaces in card
MR3179Rust infrastructure
MR3214#1219 Additional topology information in managedobject datastream
MR3225nbi: getmappings API
MR3235Interface type 'internal'
MR3297#1264 confdb: meta matchers section
MR3307#1269: Physical port to logical interface binding
MR3311installation_id and globally-unique bi_id generation
MR3312#1255 FM Pools
MR3324#1277 asset: Serial numbers checking
MR3335./noc inventory find-serial command

Improvements

MRTitle
MR2991Fix Cisco Nexus version detection
MR3006Fix Eltex.LTE.get_inventory script
MR3040managedobject datastream: admin_status field
MR3067mib: Resolve syntax references
MR3070Speedup find service on etl load.
MR3072./noc sync-mib speedup
MR3073Add extra_vars params to pm dash
MR3074./noc compile-mib is moved into ./noc mib make-collection
MR3075Allow to import MIBs in BIG5 encoding
MR3088inv.objectmodel remove renderer on protocols column
MR3099Add query param to confdb command.
MR3100Rebuild datastream when object or objectconnection change.
MR3110Add make collection from directory to mib command. Fix decode error on compile mib API.
MR3122Add iface threshold profile handler
MR3139Interface threshold handling fixes
MR3148noc/noc#1085 Confdb add VRRP. initial version. Add cisco, huawei
MR3158Add queue params to close_tt and add_comment method on TT adapter.
MR3159noc/noc#1184 IGetAlarms SA interface.
MR3166Fix ldap auth for ldap3 verison 2.0.9-2.6.1 (Py3)
MR3168Add promote_affected_tt param to AlarmEscalation config.
MR3169noc/noc#1245 Move configvalidation umbrella to 'Config | Object Policy Violation' AlarmClass.
MR3171Add escalate umbrella alarm.
MR3187Update Search by MAC in MO Inteface
MR3202#1203 Span.set_error, .set_error_from_exc helpers
MR3206Replace get_handler on classifier service.
MR3215./noc datastream rebuild --jobs
MR3217Add docs for optical model interface
MR3220Add docs for cpe part of asset model interface
MR3232#1263 bi managedobject: full link methods detalization
MR3234test snmp_metrics json
MR3236noc/noc#1087 ConfDB. add meta chassis_id
MR3238JUnit XML test report
MR3242check-labels: JUnit XML report
MR3244ConfDB: vrrp. add preempt
MR3248pymongo 3.10.1
MR3253rust project label
MR3254mongoengine 0.19.1
MR3255noc/noc#1078 datastream: config section of managedobject datastream
MR3262flake8 JUnit XML report
MR3279Reworked collection JSON sharing
MR3293inv.objectmodels collection tests
MR3295Add 'Config | Policy Violations' alarm class.
MR3304box: move asset discovery just after config one
MR3305Fix service mailsender
MR3309test collections' uuids for uniqueness
MR3313check-labels: Fix migration label detection
MR3314Retry check-labels as the labels will be fixed from webhook
MR3320Activate collections tests
MR3322add new report - load metric max
MR3323Fix report load metrics
MR3328check-labels: Detect collections label
MR3329DCS: Stop one-time resolvers
MR3330#1191 scheduler: Gracefully apply pending bulk operations on shutdown
MR3333managedobject datastream: Interface services
MR3339Catch IOError exception on beef load.
MR3353Raise error when OID tree from beef end.
MR3354Add display_hints options to get_table script method.
MR3355Restore ujson-x 1.37
MR3366#1278 sae: Pass attributes section

Bugfixes

MRTitle
MR2879Fix Exception when send message 'ServiceStub' object has no attribute 'pub'
MR3013#1186 py3: Fix request.body processing
MR3014#1185 py3: Fix error reporting on missed custom modules
MR3015Fix cleanup 0192_thresholdprofile migration.
MR3019Fix 0153_managedobjectselector_profile migration after fix 0150_managed_object_profile migration.
MR3024noc/noc#1195 Catch ValueError when convert_interface_name on TopologyDiscovery.
MR3024noc/noc#1195 Catch ValueError when convert_interface_name on TopologyDiscovery.
MR3029py3: alnum_key() for reliable sorting
MR3034Fix TypeError on http client when working over proxy.
MR3037#1193 py3: Fix MRT service
MR3041#1201 py3: Fix http client eof_mark handling
MR3045#1196 py3: Fix location handling
MR3047#1209 Fix syslogcollector/trapcollector port binding
MR3055Fix CH objectprofile datasource bool field.
MR3062noc/noc#1197 Fix encoding on interfacepath card.
MR3064py3 fix beef import encoding.
MR3066Add getting Job log to SA Monitor application
MR3068fix Alstec, Eltex pon/rg, qsw8200, dlink get_inventory
MR3069py3: Fix sync-mibs
MR3078Fix typo in text parse_table on row_wrapper param check.
MR3093fix ReportAlarmDetail
MR3094noc/noc#1188 Fix memory leak on SSH CLI.
MR3097noc/noc#1096 Fix correlator trigger handler call.
MR3104noc/noc#1213 py3. Fix TypeError when collected config convert to binary.
MR3137Fix MAC AutoAPI doc builder
MR3143Fix object datastream iter_changed managed_object check.
MR3152Fix object iter_changed_datastream typo.
MR3161Fix Segment MAC Discovery when the object links to itself
MR3162Fix interface_flap field on ReportMetrics.
MR3178Fix import __future__ print in deploy scripts
MR3184Fix promote_affected_tt param when escalation item.
MR3201sa.managedobject: Remove duplicated LAG Profile column
MR3205Fix delay escalation when start time < now time
MR3207#1227: Filter out duplicated metric measurements
MR3221Rebuild credential cache when change ManagedObjectProfile beef_storage and beef_path_template field.
MR3224py3: Fix nbi service logging
MR3230Fix typo in etl extractor.
MR3239Fix Test notification group, when id > 9.
MR3247Add ldap3 escape filter chars
MR3250#1267 Fix Peer.save()
MR3258py3: Fix whois client
MR3259#1210: Better json content type handling
MR3263Add MO queue for maintenance escalation
MR3267Fix Alarms class filter
MR3288Fix VRF type name to lowercase (were changed on interface).
MR3291Add raise_privileges and access_preference params when call login script on suggestcli discovery.
MR3303inv.objectmodels: Fix duplicated uuids
MR3327Fix typo max_serial_size on asset discovery.
MR3356py3: Fix cli.object_stream
MR3358py3: Fix name encoding when calculate config diff.
MR3359Convert cmd_next and cmd_stop to bytes on object_cli parser.
MR3371#1280 py3 ConfDB: Fix node sorting
MR3373py3: Fix adaptive prompt pattern
MR3374py3: Fix telnet sb/se logging
MR3376py3: Clean ETL loader stream from unreadable characters
MR3379Add fm_pool to field managed object loader.
MR3381py3: Fix Selectors over attributes
MR3384Fix NRI discovery

Code Cleanup

MRTitle
MR3036noc/noc#1198 Fix AlarmDiagnostic.get_diagnostics encode.
MR3053Update docs/src/en/services/icqsender.rst
MR3060Fix activator.http_get decode error.
MR3061noc/noc#1197 Fix sortdict on managedobject card.
MR3071py3. Fix MIBAPI service.
MR3102LLDP-MIB render test
MR3107Update requirements/deploy scripts for py3/py2 compat
MR3138Links to MR list
MR3195py3. Not encoding utf-8 on geocodercache.
MR3200noc/noc#1250 Fix ETL for py3.
MR3204noc/noc#1256 Fix future.set_exception_info on threadpool for py3.
MR3208Remove 20.1 deprecations
MR3209Bump pynsq version to 0.8.3
MR3216Code formatting
MR3219Fix bytes on script super_username submit.
MR3233py3: Fix bi extract
MR3249flake8: Enable W605 check
MR3251rust: remove "extern crate"
MR3252CI: Remove migrate stages
MR3260Remove deprecated models and applications
MR3261RCA neighbor cache is default and only
MR3264ujson 2.0.2
MR3285Merge back collections/ to main repo
MR3286Restore tests
MR3296flake8: Enable F403 and F405 checks
MR3310inv.connectiontypes UUID uniqueueness test
MR3315Fix cr_context on model
MR3316Replace gender for c14 connection type part1
MR3317Replace gender for c14 connection type part2
MR3318Replace gender for c14 connection type part3
MR3319Replace gender for c14 connection type part4
MR3336py3: mailsender fixes
MR3338Fix typo on asset discovery logging.
MR3340Replace StringIO to BytesIO on DetailReport for xlsxwriter work.
MR3350py3. Fix expected byte trace on interface-profile command.
MR3352py3: Convert rogue_chars to binary type
MR3357py3: Fix if between None and int type.
MR3362py3: Disable CLIPS parsers.
MR3365Clean data type on Consul config client.
MR3372py3: Replace split_alnum to alnum_key on matchers.
MR3375py3: Replace decode method to codecs on profiles.
MR3377py3: Convert addr and port to bytes on http client.
MR3378py3: Convert hash on geocodercache to text.

Profile Changes

Alcatel.7302

MRTitle
MR3089Alcatel 7302 fix port_map
MR3123fix Alcatel 7302 serial
MR3243Alcatel.7302 fix empty b_serial, b_revision. fix port_map. version 2

Alsitec.24xx

MRTitle
MR3119noc/noc#1181 Alstec.24xx. Fix unknown serial number, and devices with 4413 DeviceID.
MR3128noc/noc#1181 Alstec.24xx.get_inventory. Fallback to CLI when unknown SNMP part_no.
MR3144Alstec.24xx.get_inventory. Set script prefer to S.
MR3144Alstec.24xx.get_inventory. Set script prefer to S.

Cisco.IOS

MRTitle
MR3009Update Cisco.IOS profile to support old hardware
MR3173Cisco.IOS.get_lacp_neighbors, get_version. Small fixes encoding.
MR3185Cisco.IOS.get_lacp_neighbors. Fix rx_sys_id regex.
MR3229Cisco.IOS.get_version and get_capabilities cleanup code.
MR3229Cisco.IOS.get_version and get_capabilities cleanup code.
MR3237Cisco.IOS.get_switchport. Fix getting vlans from SNMP for python3.
MR3342Cisco.IOS.get_cdp_neighbors. Replace OID to mib name.
MR3351Cisco.IOS.get_inventory. Use matchers instead match.

DLink.DxS

MRTitle
MR3091Bypass confirmation of firmware upgrade on DGS-1100-10/ME
MR3363DLink.DxS. Fix decode on profile.
MR3367Dlink.DxS.get_interfaces. Remove decode ascii interface description.

Dahua.DH

MRTitle
MR3142Dahua.DH. Fix getting config from RVi web cam.
MR3174noc/noc#1216 Dahua.DH.get_version. Fix script on cam without PTZ module.

Eltex.MES

MRTitle
MR3155add_metrics_SensorStatus_Eltex_MES
MR3177fix_Environment_Sensor_Status_Eltex_MES

Eltex.MES24xx

MRTitle
MR3007Fix Eltex.MES24xx.get_lldp_neighbors script
MR3164fix_get_lldp_neighbors_Eltex_MES24xx

Generic

MRTitle
MR3020Generic.get_lldp_neighbors. Strip x00 symbol on SNMP string.
MR3023Generic.get_capabilities. Use cache value sysObjectID for getting EnterpriseID.
MR3023Generic.get_capabilities. Use cache value sysObjectID for getting EnterpriseID.
MR3032Generic.get_beef. Fix fsm collecting - not collected if get_version first.
MR3084Generic.get_capabilities. Fix bad enterprise id check.
MR3149Generic.get_beef. Catch SNMP error when collected beef.
MR3218py3. Message attribute not supported on Exception.
MR3348Generic.get_switchport. Fix getting vlans on SNMP.

H3C.VRP

MRTitle
MR3240H3C.VRP. Fix get_version and add SNMP scripts.
MR3300H3C.VRP.get_version. Fix regex for S3100-8TP-EI model.

Huawei.MA5600T

MRTitle
MR3022Huawei.MA5600T. Fix lstrip on profile parse_table.
MR3163noc/noc#1226 Huawei.MA5600T. Add pattern_more for MA5800-X2 platform.
MR3163noc/noc#1226 Huawei.MA5600T. Add pattern_more for MA5800-X2 platform.
MR3337Huawei. Disable CLI metrics.

Huawei.VRP

MRTitle
MR3105Huawei.VRP. Add S3328 to hw_series.
MR3113Huawei.VRP. Fix version and inventory script for CX300 platform.
MR3116noc/noc#1163 Huawei.VRP.get_inventory. Add S8504 slot count.
MR3126noc/noc#1163 Huawei.VRP. Add error_pattern_prompt and fix get_version encoding.
MR3345Huawei.VRP.get_switchport. Fix getting port vlan on SNMP.

Iskratel.MSAN

MRTitle
MR3344Fix send_on_syntax_error params for py3.

Juniper.JUNOS

MRTitle
MR3058Juniper.JUNOS.normalizer. Fix detect vrf type.
MR3256add snmp oid jnxVirtualChassisMemberMacAddBase. fix get_chassis_id Juniper_JUNOS

Qtech.QSW2800

MRTitle
MR2960Add support for stack in QSW-8200-28F-AC-DC
MR3222Qtech.QSW2800.get_capabilities. Fix not supported operation on 'NoneType' and 'int' type.
MR3343Qtech.QSW2800.get_fqdn. Fix py3 pattern encoding.

SKS.SKS

MRTitle
MR3127noc/noc#1180 SKS.SKS. Fix get_spanning_tree designated_port_id and get_inventory unknown E1 module.

Siklu.EH

MRTitle
MR3364Siklu.EH. Fix cleaned_input to binary data.

rare

MRTitle
MR3038Add NSN.hiX56xx.get_inventory script
MR3050DCN.DCWL. Fix profile for SNMP
MR3052DCN.DCWL.get_version. Fix typo on matchers.
MR3079Carelink.SWG. Fix pattern_more on profile.
MR3083Qtech.QSW8200.get_lldp_neighbors. Add ifName to port_type.
MR3108fix_get_inventory_Eltex_ESR_profile
MR3150fix_AlliedTelesis.AT9900_profile
MR3160add_get_ifindexes_AlliedTelesis.AT9900_profile
MR3167add_get_lldp_neighbors_AlliedTelesis.AT9900
MR3180add_cli_get_lldp_neighbors_AlliedTelesis.AT9900
MR3186add_get_capabilities_AlliedTelesis.AT9900
MR3196Fix get_interfaces for BDCOM.xPON profile
MR3241Fix invalid JSON
MR3245Fix get_interfaces NSN.TIMOS
MR3298fix_get_interfaces - port-channel no has iftype
MR3306NSN.TIMOS Fix get_intefaces script
MR3308Iscratel.ESCOM fix profile
MR3332Iskratel.ESCOM fix lldp neigborg script
MR3346py3. Cleanup get_lldp_neighbors script for SNMP.
MR3360fix_get_config_NSN.hiX56xx
MR3361fix_pattern_more_Huawei_VRP3
MR3369Fix Check metrics collect on DCN and Rotek profiles.
MR3370py3: Convert telnet_send_on_connect to binary.
MR3382NSN.TIMOS.get_interfaces. Fix empty mac address on tunnel interface.