NOC 20.2


In accordance to our Release Policy we’re proudly present release 20.2.

20.2 release contains of 249 bugfixes, optimisations and improvements. Refer to the Release Notes for details.


Python3 Compatibility

As it was declared 20.2 is the first Python3-only release. This allows us to clean up and optimize code base in following ways:

  • All Python2 compatibility layers are removed.
  • Key dependencies (Tornado, Django, etc) are upgraded to latest stable versions.
  • async/await are used instead @tornado.gen.coroutine in all the places.
  • Type annotations are heavily used in newly-introduced and in modified code.

Low-level network-handling code, including SNMP, RPC, HTTP-client and all CLI variants are rewritten from Tornado to Python3-native asyncio. It allows to clean up code, fix some old cryptic bugs and introduce gentler error- and timeout- handling.

NOC 20.2 is compatible with Python 3.7 and 3.8.

So we have to admit the Python3 migration Odyssey is over. It tooks over one year, consumes lots of work, sometimes with acrobatic exercises to maintain compatibility and stability. But in addition to code simplicity, stability and instrumentation we gave serious performance benefits. Our investigations on real-world installation shows that NOC 20.2 requires 10-15% less CPU, than previous Python 2 based releases.

Interface Description Topology Discovery

Meet the ifdesc topology discovery method. It is last-resort semi-automatic method of linking based on interface description analysis. ifdesc comes to resque when all other methods failed. Operator has to define a set of regular expressions to extract a neighbor information from descriptions and has to configure network equipment. In contrast to manual linking, ifdesc is configurable as common topology discovery method. Its precedence over other methods can be adjusted, so its results may be refined by more precise automatic methods (i.e. LLDP), and it can be used to refine more error-prone methods (i.e. MAC).

ifdesc is the 14th topology discovery method available out of the box.

MAC Blacklist

Costs cutting and lack of proper Q&A sometimes leads to large parties of network equipment with non-unique MAC addresses. Overall impact may vary: from insignificant, to ARP-cache poisoning, to inpredictable STP topology changes. It-also affects the topology discovery methods based on neighbor identification upon MAC address (LLDP, STP, MAC). So its safer to maintain a list of broken MAC addresses to notify personnel on possible problems.

MAC blacklist is maintained and distributed on deploy, like other collections, and can be adjusted and maintained locally and shared back to the repo.

LLDP discovery method now consults on MAC Blacklist and tries to mitigate problem by using neighbor information from additional TLVs.

Real-world networks often repeats the pattern when aggregation switch (A) is placed on PoP with redundant reserved power and connected to home access switches (B, C, D, E), which are powered from non-redundant city power grid.

graph TD;
    A --> B;
    A --> C;
    A --> D;
    A --> E;

During the maintenance on power grid, several access switches may expect power loss, while aggregation switch stays on-line as unaffected or switched to reserve power. Current topology-based RCA scheme will register each access switch failure as separate root cause alarms and will propagate them as separate trouble tickets. It may take a time for personnel to detect all those problems are related and inferred by power problems.

NOC 20.2 introduces additional Downlink Merge policy. Downlink Merge is enabled on aggregation switch (A) and allows to tie all direct children alarms together and to process and escalate them as a one failure. Network personnel will take a hint to check alarms as related together and will be notified to check the power.

Multi-Format Datastreams

Datastreams can be configured to separate views, filtering or morphing records. Views, called formats can be used to enrich datastreams with data from external sources, or to restring amount of data to be passed to external systems.

Each format can be configured to additional API Key role, allowing granular access.

Customized SNMP-response parsers

Broken SNMP implementations are in the wild. Sometimes responses are malformed, while still containing meaningful data. So we added Profile.get_snmp_response_parser() method to override default response parser to customized one when necessary.

New features

MR Title
!3418 Grafana: Add new DOM dash
!3550 Add Ignore Pattern func
!3600 #1287: Multi-Format DataStream
!3632 #1086 IfDesc topology discovery
!3653 #1301: MAC Blacklist
!3668 #1222 RCA ‘Downlink Merge’ policy
!3746 Profile.snmp_response_parser


MR Title
!3392 noc/noc#1283 Add allowDiskUse to reportdatasources.
!3407 Add combo connection type to Huawei Quidway series.
!3428 Add Combo port to Raisecom models.
!3442 http-parser 0.9.0
!3444 yield from
!3445 pytest 5.0.3
!3461 Fix some models
!3470 datastream: ManagedObject.object_profile.tags
!3476 Fix report max load metrics 2
!3490 Deduplicate Datastream Changes.
!3491 Replace imp to importlib on mib service.
!3494 Add SubscriberProfile to ETL loader
!3500 ThreadPoolExecutor: Support for context protocol
!3501 Update verify-models script
!3512 Register Datastream Changes when Service - Interface binding changes
!3516 Fix get_interfaces for DES-3200/C1
!3522 Add errors_in_sum, errors_out_sum column to reportmetrics.
!3524 fix_objectmodels_comboports_eltex_mes
!3527 nbi Path API: Specify start by OFM order_id
!3533 Bump requirements versions
!3539 Django 3.0.5
!3542 Add iface_hints to RemoteSystem portmapper.
!3553 Tornado 6 support
!3554 NetworkX 2.4
!3555 Make Service.subscribe a coroutine
!3557 Cleanup interface service field when remove service.
!3560 Add pool to reportdiscoverypoison.
!3561 Impoved logging when use neighbors discovery cache.
!3571 Python 3.8 tests, pyproj 2.6.0
!3580 Add quiet params to etl extract command. Supress exception if set.
!3581 Add retry escalation to Escalation limit exceeded.
!3587 add_new_model_Nag
!3588 fix_objectmodels_eltex_LTP-LTE
!3591 Add rules to test inv.objectmodels
!3592 Fix some models
!3615 Native PeriodicCallback and PeriodicOffsetCallback
!3617 http client: Asynchronous DNS resolver
!3619 Add cutomized dashboard to PM.
!3624 MAC Segment discovery. Check object level on same segment for uplink detect.
!3625 #1298 PrefixDiscovery takes account of MO’s VRF
!3626 #1299 Hide hidden custom fields from ipam view
!3627 Django 3.0.6, mongoengine 0.20.0, crontab 0.22.6, redus-3.5.2, hiredis=1.0.1, ipython 7.14.0
!3633 Add ELtex MIB preference.
!3636 Add Eltex MIBs.
!3637 Add catch OIDCollision to mib upload command. Add detect some errors when upload mib.
!3641 IOLoop.add_callback() cleanup
!3650 Native asyncio SNMP implementation
!3657 Add Huawei.MA5600T profile check rules
!3662 Add Eltex MIBs.
!3674 Add json_unique_fields to ProfileCheckRule.
!3678 Add ‘Discards In’ and ‘Discards Out’ to Report metrics.
!3682 noc/noc#1295 Add Cisco ASR1001-X model.
!3685 Remove Service.ioloop
!3694 Streamlined SNMP Beef implementation
!3695 asyncio-native NSQ topic implementation
!3705 asyncio-native CLI
!3713 Replace resolve_ifindexes to resolve_properties on InterfaceDiscovery.
!3716 Fix filling Capabilities from ConfDB.
!3719 Add dump argument to confdb commands.
!3747 Add BI Dashboard collection.
!3751 Raise CLI CLILowPrivileges when no up privileges by enable.
!3752 Send command_exit when close cli stream.
!3754 Add get_fqdn to profiles.
!3760 Add ‘Exclude on MAC Black List’ param to ReportDiscovery Poison.
!3763 Add raw_result params to script http methods.
!3765 Add errors=“replace” policy to script parse_object_stream.
!3768 Change docker image’s python to 3


MR Title
!3389 fix report - load metric max
!3395 Fix save on ModelProtectionProfile.
!3397 Tgsender: fix py3
!3400 Fix bi encode error when empty field description.
!3405 Fix changing objectmodel combo field.
!3419 Fix AlowDiskUse param on aggregate report
!3426 Fix DatasourceCache compress encoding.
!3440 Fix Consul session lost logging
!3446 Disable smart_text over return http.get json result.
!3448 Fix comma on mo pm dashboard.
!3479 py3: Fix DNS RR comparison
!3483 Add is_managed=False to maintenance escalator handler.
!3484 Fix detect selected column on ObjectDetail Report.
!3502 Fix bytes on asset generate_serial.
!3503 Change enable_link to enable_service on nri service discovery.
!3507 py3: Fix whois update-cache
!3513 Add json_unique_fields to mibpreference.
!3517 Fix scheduler shutdown
!3518 Fix CH Dictionary split data to chunk.
!3534 Remove validation policy button from UI.
!3537 Fix fix-pip on PIP’s beta versions
!3556 Fix SAE return redirect. Was broken on noc/noc!3508.
!3574 Fix Link.managed_objects on Django3
!3576 Fix not operand on ConfDB.
!3612 Catch portmapper exception on NRI discovery.
!3618 httpclient: Fix EOF handling
!3620 noc/noc#1283 Fix ReportLinkDetail for py3.
!3634 Fix trace when export fm.mibpreference collection by command.
!3640 Fix detect coroutine on Script.on_error_sequence.
!3646 Fix ReportDiscovery Poison MAC view.
!3649 Fix script on_unprivileged_prompt when no raise privileges.
!3651 classifier: Fix bytes processing
!3664 Fix ‘TypeError’ on confdb iter_nodes sorted.
!3666 HTTPClient: Better handling of connection errors
!3671 chwriter: Fix channel messages calculation
!3672 Fix race condition during lazy mib loading
!3673 Lock all RPC deserialization for a better times
!3676 Fix logger argument on xmac discovery.
!3692 Fix dictionary keys changed during iteration on extdocapplication.
!3693 SNMP: Fix GETNEXT raw_varbinds processing
!3702 Fix confdb protocols applicators for type normalizer.
!3707 Fix MRT service
!3708 ETL: Fix parallel extraction for Oracle
!3710 Fix CLI loop context usage
!3712 redis: delete_many may be called with empty keys
!3714 Fix typo in InterfaceDiscovery resolve_properties.
!3724 httpclient: Fix connection reset handling
!3729 CLI: Fix keepalive timeout handling
!3731 py3. Fix get_colors.
!3749 CLI: Fix connection reset handling
!3750 Fix typo on MRT service.
!3753 Fix params on LLDP Discovery logger.
!3756 snmp: Fix broken error-index
!3757 Fix Script.on_error_sequence sequence type.
!3758 Fix typo on MacBlackList js Application.
!3759 Fix NSQ topic shutdown sequence
!3762 Fix BER unknown types reporting
!3769 Fix script.submit debug.
!3770 Fix catch ConnectionResetError on http_client.
!3771 Right fix ConnectionResetError on http client.

Code Cleanup

MR Title
!3347 py3. Change rogue_chars on profiles to bin.
!3387 Remove py2 tests
!3388 Set black target to python 3.6
!3390 Fix DDash sensorstatus
!3390 Fix DDash sensorstatus
!3394 Remove CLIPS config validation
!3401 py3. Fix compile-bytecode.
!3417 py3: Remove six.PY2/six.PY3 usage
!3421 py3: Remove six type aliases usage
!3439 py3: Remove six.iter* wrappers
!3447 Remove @six.python_2_unicode_compatible and six.with_metaclass
!3457 Move to py3-style type annotations
!3458 Clean up .docker directory
!3463 Remove six
!3464 Cleanup noc.core.comp bord/bchr/make_bytes functions
!3465 Use dict comprehensions
!3466 Use py3 to build docs
!3467 Fix docs builder
!3469 py3: Fix
!3473 Remove CLIPS-based validation models
!3474 mypy: Remove –py2 flag
!3508 py3: Remote raise tornado.gen.Return() usage
!3509 Fix type annotations
!3519 Source coding specification cleanup
!3523 py3.8 compatibility fixes
!3525 Use native time.perf_counter()
!3526 Remove from builtins import
!3528 Remove pypy support
!3529 Ping: Remove io_loop parameter
!3531 httpclient: Remove io_loop parameter
!3535 DCS: Remove io_loop parameter
!3536 Remove coding directive
!3538 Remove deprecated get_absolute_url method
!3540 Remove io_loop parameter from UDPServer
!3541 Remove io_loop parameter from CLI
!3543 Remove PeriodicOffsetCallback and NSQ TopicQueue io_loop parameter
!3544 Remove snmp io_loop parameter
!3545 Remove deprecated TracebackFuture alias
!3546 Remove scheduler IOLoop parameter
!3593 Cleanup super() call
!3606 Cleanup tornado.gen usage
!3614 Fix DOM ddash
!3628 Remove deprecated get_future_loop() compatibility call
!3631 Fix import deprecation warning
!3639 Fix id ddash for Grafana 6
!3643 Add/new ddash stp delta
!3659 Fix Radio Ddash
!3665 Replace socket.error with OSError
!3726 20.2 deprecations

Profile Changes


MR Title
!3594 Alcatel.7302. Add SNMP to profile.


MR Title
!3437 noc/noc#1181 Alstec.24xx. Add platform name normalizer.


MR Title
!3767 Cisco.IOS.get_interfaces. Use dict structure instead list.
MR Title
!3596 DLink.DxS_Cisco_CLI.get_interfaces. Fix description decode.
!3727 DLink.DxS_Smart.get_interfaces. Fix typo.
!3737 DLink.DxS.get_lldp_neighbors. py3. Fix remote_port format to string.


MR Title
!3391 Eltex.MES Update metrics
!3434 Eltex.MES: Fix get_mac_address_table
!3441 Eltex.MES: Update profile for 3124/3124F
!3487 Eltex.MES. Add detected platform to profile.
!3547 add_new_model_eltex_mes-3116F
!3679 Eltex.MES.get_capabilities. Add SNMP methods.
!3680 Eltex.MES.get_interfaces. Use Generic SNMP method.
!3700 Eltex.MES.get_interfaces. Add Vlan and Lo filter on SNMP.
!3722 Eltex.MES. Add always_prefer S for 33243348 models.


MR Title
!3697 Eltex.MES24xx. Add config prompt to profile.
!3738 Eltex.MES24xx.get_lldp_neighbors. Add ‘Interface Alias’ to port subtype.


MR Title
!3559 Generic.get_chassis_id. Add 00:01:02:03:04:05 to IGNORED_MACS.
!3565 Generic.get_mac_address_table. Filter vlan 0 number.
!3573 Generic.get_interfaces. Refactor script for get_interface_properties use.
!3648 Add Generic.get_fqdn script.
!3681 Eltex.MES.get_spanning_tree. Add SNMP method.
!3687 Generic.get_beef. Fix encode method.
!3688 Generic.get_spanning_tree. Add detect port role.
!3696 Generic.get_intefaces. Filter equal values on getting portchannel.
!3706 Add get_interface_properties to get_ifindexes profiles
!3733 Generic.get_vlans. Fix py3.
!3734 Generic.get_mac_address_table. Skip unknown interfaces.


MR Title
!3412 Huawei.MA5600T.get_cpe. Use codecs.encode instead .encode.
!3582 Huawei.MA5600T.get_interfaces. Add interface hints field.
!3595 Huawei.MA5600T. Add get_lldp_neighbors script.
!3654 Huawei.MA5600T.get_portchannel. Add allow_empty_response to ‘display lacp’ command.
!3656 Huawei.MA5600T.get_portchannel. Use allow_empty_response on ‘display lacp’ command.


MR Title
!3669 Fix tagged vlan parsing on switch with 40G interfaces
!3699 Huawei.VRP.get_interfaces. Use show vlan command for switchport info.
!3718 Huawei.VRP. Fixes ConfDB normalizer.


MR Title
!3755 noc/noc#1232 Iskrate.MSAN. Change detect version to ‘Steer Version’.


MR Title
!3386 Fix Juniper.JUNOS.get_interfaces script
!3413 Juniper.JUNOS.get_interfaces. Use codecs.decode insterad .decode.


MR Title
!3468 fix get_version - space removed in platform
!3492 noc/noc#1230 Orion.NOS. Add convert_interface_name method.
!3590 Fix Orion.NOS.get_lldp_neighbors script


MR Title
!3599 noc/noc#1231 Qtech.QSW.get_version. Fix version regex.


MR Title
!3711 noc/noc#1190 Qtech.QSW2800.get_version. Fix 3.0.1-R1-BETA3 firmware.
!3721 Qtech.QSW2800. Fix patter_error for QSW-3470-10T with 3.0.1-R1-BETA3.
!3766 noc/noc#1190 Qtech.QSW2800.get_lldp_neighbors. Add port description and sysname


MR Title
!3399 Raisecom.ROS.get_version. Add detect Qtech vendor.
!3566 Raisecom.ROS. Add fastethernet1/0/1 to management for ISCOM26(?:24
!3735 Raisecom.ROS. Add SNMP to scripts.
!3736 Raisecom.ROS.get_capabilities. Fix STP detect.


MR Title
!3601 noc/noc#1180 SKS.SKS.get_interfaces. Fix tagged vlan none on command.


MR Title
!3610 Siklu.EH.get_mac_address_table. Fix mac type detected if quota column end.


MR Title
!3689 noc/noc#1290 Zyxel.ZyNOS. py3, Convert pattern_zynos prompt to binary.
!3720 Zyxel.ZynOS.get_inventory. Disable keep_cli_session after script.


MR Title
!3393 Eltex.MES5448 update profile
!3410 py3. Cleanup OID iface MAC to IF-MIB::ifPhysAddress name.
!3420 Generic.get_mac_address_table. Fix getting mac address tables by SNMP.
!3450 DCN.DCWL.get_metrics. Fix empty mac address.
!3453 Convert command_submit on self.cli to bytes.
!3454 Qtech.QSW.get_chassis_id. Add render_mac to SNMP method.
!3459 py3. Fix convert hex values on profiles.
!3495 noc/noc#1175 Huawei.MA5300.get_version. Add MA5303 to detecting profile.
!3725 HP.ProCurve. Cleanup py3.
!3739 DCN.DCWL.get_metrics. Fix rtt check.
!3748 Qtech.BFC_PBIC_S. Use strict snmp response parser.
!3764 Vitesse.VSC. Fix profile pattern_more.