In accordance to our Release Policy
we’re proudly present release 20.2.
20.2 release contains of 249 bugfixes, optimisations and improvements.
Refer to the Release Notes
for details.
Highlights
Python3 Compatibility
As it was declared 20.2 is the first Python3-only release. This allows us to clean up
and optimize code base in following ways:
- All Python2 compatibility layers are removed.
- Key dependencies (Tornado, Django, etc) are upgraded to latest stable versions.
- async/await are used instead
@tornado.gen.coroutine in all the places.
- Type annotations are heavily used in newly-introduced and in modified code.
Low-level network-handling code, including SNMP, RPC, HTTP-client and
all CLI variants are rewritten from Tornado to Python3-native asyncio.
It allows to clean up code, fix some old cryptic bugs and introduce
gentler error- and timeout- handling.
NOC 20.2 is compatible with Python 3.7 and 3.8.
So we have to admit the Python3 migration Odyssey is over. It tooks
over one year, consumes lots of work, sometimes with acrobatic exercises
to maintain compatibility and stability. But in addition to code
simplicity, stability and instrumentation we gave serious performance benefits. Our
investigations on real-world installation shows that NOC 20.2 requires
10-15% less CPU, than previous Python 2 based releases.
Interface Description Topology Discovery
Meet the ifdesc topology discovery method. It is
last-resort semi-automatic method of linking based on interface
description analysis. ifdesc comes to resque when all other
methods failed. Operator has to define a set of regular expressions
to extract a neighbor information from descriptions and has to
configure network equipment. In contrast to manual linking,
ifdesc is configurable as common topology discovery method.
Its precedence over other methods can be adjusted, so its results
may be refined by more precise automatic methods (i.e. LLDP), and it can be
used to refine more error-prone methods (i.e. MAC).
ifdesc is the 14th topology discovery method available out of the box.
MAC Blacklist
Costs cutting and lack of proper Q&A sometimes leads to large parties
of network equipment with non-unique MAC addresses. Overall
impact may vary: from insignificant, to ARP-cache poisoning, to inpredictable
STP topology changes. It-also affects the topology discovery methods
based on neighbor identification upon MAC address (LLDP, STP, MAC).
So its safer to maintain a list of broken MAC addresses to notify
personnel on possible problems.
MAC blacklist is maintained and distributed on deploy, like other
collections, and can be adjusted and maintained locally and shared back
to the repo.
LLDP discovery method now consults on MAC Blacklist and tries to mitigate
problem by using neighbor information from additional TLVs.
RCA Downlink Merge Policy
Real-world networks often repeats the pattern when
aggregation switch (A) is placed on PoP with redundant reserved power
and connected to home access switches (B, C, D, E), which are
powered from non-redundant city power grid.
graph TD;
A --> B;
A --> C;
A --> D;
A --> E;
During the maintenance on power grid, several access switches may
expect power loss, while aggregation switch stays on-line as
unaffected or switched to reserve power. Current topology-based RCA
scheme will register each access switch failure as separate root cause
alarms and will propagate them as separate trouble tickets. It may
take a time for personnel to detect all those problems are related
and inferred by power problems.
NOC 20.2 introduces additional Downlink Merge policy. Downlink Merge
is enabled on aggregation switch (A) and allows to tie all direct
children alarms together and to process and escalate them as a one failure.
Network personnel will take a hint to check alarms as related together
and will be notified to check the power.
Datastreams can be configured to separate views, filtering or morphing
records. Views, called formats can be used to enrich datastreams
with data from external sources, or to restring amount of data
to be passed to external systems.
Each format can be configured to additional API Key role, allowing
granular access.
Customized SNMP-response parsers
Broken SNMP implementations are in the wild. Sometimes responses
are malformed, while still containing meaningful data. So we added
Profile.get_snmp_response_parser() method to override default
response parser to customized one when necessary.
New features
| MR |
Title |
| !3418 |
Grafana: Add new DOM dash |
| !3550 |
Add Ignore Pattern func |
| !3600 |
#1287: Multi-Format DataStream |
| !3632 |
#1086 IfDesc topology discovery |
| !3653 |
#1301: MAC Blacklist |
| !3668 |
#1222 RCA ‘Downlink Merge’ policy |
| !3746 |
Profile.snmp_response_parser |
Improvements
| MR |
Title |
| !3392 |
noc/noc#1283 Add allowDiskUse to reportdatasources. |
| !3407 |
Add combo connection type to Huawei Quidway series. |
| !3428 |
Add Combo port to Raisecom models. |
| !3442 |
http-parser 0.9.0 |
| !3444 |
yield from |
| !3445 |
pytest 5.0.3 |
| !3461 |
Fix some models |
| !3470 |
datastream: ManagedObject.object_profile.tags |
| !3476 |
Fix report max load metrics 2 |
| !3490 |
Deduplicate Datastream Changes. |
| !3491 |
Replace imp to importlib on mib service. |
| !3494 |
Add SubscriberProfile to ETL loader |
| !3500 |
ThreadPoolExecutor: Support for context protocol |
| !3501 |
Update verify-models script |
| !3512 |
Register Datastream Changes when Service - Interface binding changes |
| !3516 |
Fix get_interfaces for DES-3200/C1 |
| !3522 |
Add errors_in_sum, errors_out_sum column to reportmetrics. |
| !3524 |
fix_objectmodels_comboports_eltex_mes |
| !3527 |
nbi Path API: Specify start by OFM order_id |
| !3533 |
Bump requirements versions |
| !3539 |
Django 3.0.5 |
| !3542 |
Add iface_hints to RemoteSystem portmapper. |
| !3553 |
Tornado 6 support |
| !3554 |
NetworkX 2.4 |
| !3555 |
Make Service.subscribe a coroutine |
| !3557 |
Cleanup interface service field when remove service. |
| !3560 |
Add pool to reportdiscoverypoison. |
| !3561 |
Impoved logging when use neighbors discovery cache. |
| !3571 |
Python 3.8 tests, pyproj 2.6.0 |
| !3580 |
Add quiet params to etl extract command. Supress exception if set. |
| !3581 |
Add retry escalation to Escalation limit exceeded. |
| !3587 |
add_new_model_Nag |
| !3588 |
fix_objectmodels_eltex_LTP-LTE |
| !3591 |
Add rules to test inv.objectmodels |
| !3592 |
Fix some models |
| !3615 |
Native PeriodicCallback and PeriodicOffsetCallback |
| !3617 |
http client: Asynchronous DNS resolver |
| !3619 |
Add cutomized dashboard to PM. |
| !3624 |
MAC Segment discovery. Check object level on same segment for uplink detect. |
| !3625 |
#1298 PrefixDiscovery takes account of MO’s VRF |
| !3626 |
#1299 Hide hidden custom fields from ipam view |
| !3627 |
Django 3.0.6, mongoengine 0.20.0, crontab 0.22.6, redus-3.5.2, hiredis=1.0.1, ipython 7.14.0 |
| !3633 |
Add ELtex MIB preference. |
| !3636 |
Add Eltex MIBs. |
| !3637 |
Add catch OIDCollision to mib upload command. Add detect some errors when upload mib. |
| !3641 |
IOLoop.add_callback() cleanup |
| !3650 |
Native asyncio SNMP implementation |
| !3657 |
Add Huawei.MA5600T profile check rules |
| !3662 |
Add Eltex MIBs. |
| !3674 |
Add json_unique_fields to ProfileCheckRule. |
| !3678 |
Add ‘Discards In’ and ‘Discards Out’ to Report metrics. |
| !3682 |
noc/noc#1295 Add Cisco ASR1001-X model. |
| !3685 |
Remove Service.ioloop |
| !3694 |
Streamlined SNMP Beef implementation |
| !3695 |
asyncio-native NSQ topic implementation |
| !3705 |
asyncio-native CLI |
| !3713 |
Replace resolve_ifindexes to resolve_properties on InterfaceDiscovery. |
| !3716 |
Fix filling Capabilities from ConfDB. |
| !3719 |
Add dump argument to confdb commands. |
| !3747 |
Add BI Dashboard collection. |
| !3751 |
Raise CLI CLILowPrivileges when no up privileges by enable. |
| !3752 |
Send command_exit when close cli stream. |
| !3754 |
Add get_fqdn to profiles. |
| !3760 |
Add ‘Exclude on MAC Black List’ param to ReportDiscovery Poison. |
| !3763 |
Add raw_result params to script http methods. |
| !3765 |
Add errors=“replace” policy to script parse_object_stream. |
| !3768 |
Change docker image’s python to 3 |
Bugfixes
| MR |
Title |
| !3389 |
fix report - load metric max |
| !3395 |
Fix save on ModelProtectionProfile. |
| !3397 |
Tgsender: fix py3 |
| !3400 |
Fix bi encode error when empty field description. |
| !3405 |
Fix changing objectmodel combo field. |
| !3419 |
Fix AlowDiskUse param on aggregate report |
| !3426 |
Fix DatasourceCache compress encoding. |
| !3440 |
Fix Consul session lost logging |
| !3446 |
Disable smart_text over return http.get json result. |
| !3448 |
Fix comma on mo pm dashboard. |
| !3479 |
py3: Fix DNS RR comparison |
| !3483 |
Add is_managed=False to maintenance escalator handler. |
| !3484 |
Fix detect selected column on ObjectDetail Report. |
| !3502 |
Fix bytes on asset generate_serial. |
| !3503 |
Change enable_link to enable_service on nri service discovery. |
| !3507 |
py3: Fix whois update-cache |
| !3513 |
Add json_unique_fields to mibpreference. |
| !3517 |
Fix scheduler shutdown |
| !3518 |
Fix CH Dictionary split data to chunk. |
| !3534 |
Remove validation policy button from UI. |
| !3537 |
Fix fix-pip on PIP’s beta versions |
| !3556 |
Fix SAE return redirect. Was broken on noc/noc!3508. |
| !3574 |
Fix Link.managed_objects on Django3 |
| !3576 |
Fix not operand on ConfDB. |
| !3612 |
Catch portmapper exception on NRI discovery. |
| !3618 |
httpclient: Fix EOF handling |
| !3620 |
noc/noc#1283 Fix ReportLinkDetail for py3. |
| !3634 |
Fix trace when export fm.mibpreference collection by command. |
| !3640 |
Fix detect coroutine on Script.on_error_sequence. |
| !3646 |
Fix ReportDiscovery Poison MAC view. |
| !3649 |
Fix script on_unprivileged_prompt when no raise privileges. |
| !3651 |
classifier: Fix bytes processing |
| !3664 |
Fix ‘TypeError’ on confdb iter_nodes sorted. |
| !3666 |
HTTPClient: Better handling of connection errors |
| !3671 |
chwriter: Fix channel messages calculation |
| !3672 |
Fix race condition during lazy mib loading |
| !3673 |
Lock all RPC deserialization for a better times |
| !3676 |
Fix logger argument on xmac discovery. |
| !3692 |
Fix dictionary keys changed during iteration on extdocapplication. |
| !3693 |
SNMP: Fix GETNEXT raw_varbinds processing |
| !3702 |
Fix confdb protocols applicators for type normalizer. |
| !3707 |
Fix MRT service |
| !3708 |
ETL: Fix parallel extraction for Oracle |
| !3710 |
Fix CLI loop context usage |
| !3712 |
redis: delete_many may be called with empty keys |
| !3714 |
Fix typo in InterfaceDiscovery resolve_properties. |
| !3724 |
httpclient: Fix connection reset handling |
| !3729 |
CLI: Fix keepalive timeout handling |
| !3731 |
py3. Fix get_colors. |
| !3749 |
CLI: Fix connection reset handling |
| !3750 |
Fix typo on MRT service. |
| !3753 |
Fix params on LLDP Discovery logger. |
| !3756 |
snmp: Fix broken error-index |
| !3757 |
Fix Script.on_error_sequence sequence type. |
| !3758 |
Fix typo on MacBlackList js Application. |
| !3759 |
Fix NSQ topic shutdown sequence |
| !3762 |
Fix BER unknown types reporting |
| !3769 |
Fix script.submit debug. |
| !3770 |
Fix catch ConnectionResetError on http_client. |
| !3771 |
Right fix ConnectionResetError on http client. |
Code Cleanup
| MR |
Title |
| !3347 |
py3. Change rogue_chars on profiles to bin. |
| !3387 |
Remove py2 tests |
| !3388 |
Set black target to python 3.6 |
| !3390 |
Fix DDash sensorstatus |
| !3390 |
Fix DDash sensorstatus |
| !3394 |
Remove CLIPS config validation |
| !3401 |
py3. Fix compile-bytecode. |
| !3417 |
py3: Remove six.PY2/six.PY3 usage |
| !3421 |
py3: Remove six type aliases usage |
| !3439 |
py3: Remove six.iter* wrappers |
| !3447 |
Remove @six.python_2_unicode_compatible and six.with_metaclass |
| !3457 |
Move to py3-style type annotations |
| !3458 |
Clean up .docker directory |
| !3463 |
Remove six |
| !3464 |
Cleanup noc.core.comp bord/bchr/make_bytes functions |
| !3465 |
Use dict comprehensions |
| !3466 |
Use py3 to build docs |
| !3467 |
Fix docs builder |
| !3469 |
py3: Fix build-go.py |
| !3473 |
Remove CLIPS-based validation models |
| !3474 |
mypy: Remove –py2 flag |
| !3508 |
py3: Remote raise tornado.gen.Return() usage |
| !3509 |
Fix type annotations |
| !3519 |
Source coding specification cleanup |
| !3523 |
py3.8 compatibility fixes |
| !3525 |
Use native time.perf_counter() |
| !3526 |
Remove from builtins import |
| !3528 |
Remove pypy support |
| !3529 |
Ping: Remove io_loop parameter |
| !3531 |
httpclient: Remove io_loop parameter |
| !3535 |
DCS: Remove io_loop parameter |
| !3536 |
Remove coding directive |
| !3538 |
Remove deprecated get_absolute_url method |
| !3540 |
Remove io_loop parameter from UDPServer |
| !3541 |
Remove io_loop parameter from CLI |
| !3543 |
Remove PeriodicOffsetCallback and NSQ TopicQueue io_loop parameter |
| !3544 |
Remove snmp io_loop parameter |
| !3545 |
Remove deprecated TracebackFuture alias |
| !3546 |
Remove scheduler IOLoop parameter |
| !3593 |
Cleanup super() call |
| !3606 |
Cleanup tornado.gen usage |
| !3614 |
Fix DOM ddash |
| !3628 |
Remove deprecated get_future_loop() compatibility call |
| !3631 |
Fix import deprecation warning |
| !3639 |
Fix id ddash for Grafana 6 |
| !3643 |
Add/new ddash stp delta |
| !3659 |
Fix Radio Ddash |
| !3665 |
Replace socket.error with OSError |
| !3726 |
20.2 deprecations |
Profile Changes
Alcatel.7302
| MR |
Title |
| !3594 |
Alcatel.7302. Add SNMP to profile. |
Alsitec.24xx
| MR |
Title |
| !3437 |
noc/noc#1181 Alstec.24xx. Add platform name normalizer. |
Cisco.IOS
| MR |
Title |
| !3767 |
Cisco.IOS.get_interfaces. Use dict structure instead list. |
DLink.DxS
| MR |
Title |
| !3596 |
DLink.DxS_Cisco_CLI.get_interfaces. Fix description decode. |
| !3727 |
DLink.DxS_Smart.get_interfaces. Fix typo. |
| !3737 |
DLink.DxS.get_lldp_neighbors. py3. Fix remote_port format to string. |
Eltex.MES
| MR |
Title |
| !3391 |
Eltex.MES Update metrics |
| !3434 |
Eltex.MES: Fix get_mac_address_table |
| !3441 |
Eltex.MES: Update profile for 3124/3124F |
| !3487 |
Eltex.MES. Add detected platform to profile. |
| !3547 |
add_new_model_eltex_mes-3116F |
| !3679 |
Eltex.MES.get_capabilities. Add SNMP methods. |
| !3680 |
Eltex.MES.get_interfaces. Use Generic SNMP method. |
| !3700 |
Eltex.MES.get_interfaces. Add Vlan and Lo filter on SNMP. |
| !3722 |
Eltex.MES. Add always_prefer S for 3324⁄3348 models. |
Eltex.MES24xx
| MR |
Title |
| !3697 |
Eltex.MES24xx. Add config prompt to profile. |
| !3738 |
Eltex.MES24xx.get_lldp_neighbors. Add ‘Interface Alias’ to port subtype. |
Generic
| MR |
Title |
| !3559 |
Generic.get_chassis_id. Add 00:01:02:03:04:05 to IGNORED_MACS. |
| !3565 |
Generic.get_mac_address_table. Filter vlan 0 number. |
| !3573 |
Generic.get_interfaces. Refactor script for get_interface_properties use. |
| !3648 |
Add Generic.get_fqdn script. |
| !3681 |
Eltex.MES.get_spanning_tree. Add SNMP method. |
| !3687 |
Generic.get_beef. Fix encode method. |
| !3688 |
Generic.get_spanning_tree. Add detect port role. |
| !3696 |
Generic.get_intefaces. Filter equal values on getting portchannel. |
| !3706 |
Add get_interface_properties to get_ifindexes profiles |
| !3733 |
Generic.get_vlans. Fix py3. |
| !3734 |
Generic.get_mac_address_table. Skip unknown interfaces. |
Huawei.MA5600T
| MR |
Title |
| !3412 |
Huawei.MA5600T.get_cpe. Use codecs.encode instead .encode. |
| !3582 |
Huawei.MA5600T.get_interfaces. Add interface hints field. |
| !3595 |
Huawei.MA5600T. Add get_lldp_neighbors script. |
| !3654 |
Huawei.MA5600T.get_portchannel. Add allow_empty_response to ‘display lacp’ command. |
| !3656 |
Huawei.MA5600T.get_portchannel. Use allow_empty_response on ‘display lacp’ command. |
Huawei.VRP
| MR |
Title |
| !3669 |
Fix tagged vlan parsing on switch with 40G interfaces |
| !3699 |
Huawei.VRP.get_interfaces. Use show vlan command for switchport info. |
| !3718 |
Huawei.VRP. Fixes ConfDB normalizer. |
Iskratel.MSAN
| MR |
Title |
| !3755 |
noc/noc#1232 Iskrate.MSAN. Change detect version to ‘Steer Version’. |
Juniper.JUNOS
| MR |
Title |
| !3386 |
Fix Juniper.JUNOS.get_interfaces script |
| !3413 |
Juniper.JUNOS.get_interfaces. Use codecs.decode insterad .decode. |
Orion.NOS
| MR |
Title |
| !3468 |
fix get_version - space removed in platform |
| !3492 |
noc/noc#1230 Orion.NOS. Add convert_interface_name method. |
| !3590 |
Fix Orion.NOS.get_lldp_neighbors script |
Qtech.QSW
| MR |
Title |
| !3599 |
noc/noc#1231 Qtech.QSW.get_version. Fix version regex. |
Qtech.QSW2800
| MR |
Title |
| !3711 |
noc/noc#1190 Qtech.QSW2800.get_version. Fix 3.0.1-R1-BETA3 firmware. |
| !3721 |
Qtech.QSW2800. Fix patter_error for QSW-3470-10T with 3.0.1-R1-BETA3. |
| !3766 |
noc/noc#1190 Qtech.QSW2800.get_lldp_neighbors. Add port description and sysname |
| MR |
Title |
| !3399 |
Raisecom.ROS.get_version. Add detect Qtech vendor. |
| !3566 |
Raisecom.ROS. Add fastethernet1/0/1 to management for ISCOM26(?:24 |
| !3735 |
Raisecom.ROS. Add SNMP to scripts. |
| !3736 |
Raisecom.ROS.get_capabilities. Fix STP detect. |
SKS.SKS
| MR |
Title |
| !3601 |
noc/noc#1180 SKS.SKS.get_interfaces. Fix tagged vlan none on command. |
Siklu.EH
| MR |
Title |
| !3610 |
Siklu.EH.get_mac_address_table. Fix mac type detected if quota column end. |
Zyxel.ZyNOS
| MR |
Title |
| !3689 |
noc/noc#1290 Zyxel.ZyNOS. py3, Convert pattern_zynos prompt to binary. |
| !3720 |
Zyxel.ZynOS.get_inventory. Disable keep_cli_session after script. |
rare
| MR |
Title |
| !3393 |
Eltex.MES5448 update profile |
| !3410 |
py3. Cleanup OID iface MAC to IF-MIB::ifPhysAddress name. |
| !3420 |
Generic.get_mac_address_table. Fix getting mac address tables by SNMP. |
| !3450 |
DCN.DCWL.get_metrics. Fix empty mac address. |
| !3453 |
Convert command_submit on self.cli to bytes. |
| !3454 |
Qtech.QSW.get_chassis_id. Add render_mac to SNMP method. |
| !3459 |
py3. Fix convert hex values on profiles. |
| !3495 |
noc/noc#1175 Huawei.MA5300.get_version. Add MA5303 to detecting profile. |
| !3725 |
HP.ProCurve. Cleanup py3. |
| !3739 |
DCN.DCWL.get_metrics. Fix rtt check. |
| !3748 |
Qtech.BFC_PBIC_S. Use strict snmp response parser. |
| !3764 |
Vitesse.VSC. Fix profile pattern_more. |
