NOC 22.2¶
22.2 release contains 286 bugfixes, optimisations and improvements.
Highlights¶
Gufo Labs Collaboration¶
Gufo Labs is a private IT-consulting company based in Milan, Italy. Among other things, they focus on extracting the vital parts of the NOC technologies, reworking them, and releasing them as separate open-source packages. Separation into independent packages allows a focus on API, documentation, reliability, portability, resource usage, and performance.
Known as Gufo Stack, this software is the valuable tool not only for NOC but every network-related engineer and developer.
ETL¶
ETL system got the ability to fetch data incrementally. Depending on the remote system, ETL can request only changed data. It reduces imposed load and allows reducing of the synchronization intervals.
Additionally, ETL now can extract and load Managed Object's capabilities.
MX¶
MX service got the ability to route and process messages based on labels. To reduce the
system resoure usage, parts of mx service can be integrated directly in the producers'
processes.
The Fault Management got the ability to mirror the messages to the mx, allowing
real-time export to the Kafka. syslogcollector can mirror incoming syslog messages,
trapcollector can mirror semi-parsed SNMP Trap PDUs, while classifier
can mirror the classified messages. This greatly increases the NOC's capability
to pump the Data Lakes and all the other's "Big Data" stuff.
Metrics service¶
After several profiling sessions, the metrics service requires much less CPU and memory.
Ping¶
NOC has switched to the Gufo Ping. Gufo Ping is the
successor of the NOC's Cython-accelerated ping implementation. Compared to the previous
implementation it has distinctive features:
- Implemented in Rust language using pyo3 bindings.
- Memory and CPU optimized.
- 100% test coverage.
- Built-in benchmarks.
- Reduced context switches.
- Reduced python-to-binary switches.
- Applies BPF filter to separate concurrent ping processes.
- Packaged as binary wheels for the major platforms.
Python¶
Python 3.7 support is deprecated, leaving support for 3.8-3.10.
Internals¶
tornadohas been removed from all critical paths. Next version will remove explicittornadorequirements.
New features¶
| MR | Title |
|---|---|
| MR6264 | noc/noc#1780 Incremental ETL |
| MR6422 | noc/noc#1861 Add send snmptrap message to mx. |
| MR6433 | noc/noc#1836 Add StreamingConfig param to Activator.script RPC call. |
| MR6437 | noc/noc#1351 Add capabilities to ManagedObject ETL extractor. |
| MR6442 | noc/noc#1861 Add send syslog message to mx. |
| MR6451 | noc/noc#1838 Add event to mx message. |
| MR6506 | noc/noc#1857 Add CredentialCheckRule model. |
| MR6509 | noc/noc#1853 Add diagnostics to ManagedObject. |
| MR6530 | noc/noc#534 Add Diagnostic discovery. |
| MR6556 | noc/noc#534 Add User-defined ObjectDiagnosticConfig |
| MR6669 | noc/noc#1907 Move Message Router to core. |
Improvements¶
| MR | Title |
|---|---|
| MR5084 | noc/noc#1820 Add hostname_id to DiscoveryID model |
| MR6211 | Implement message storm protection |
| MR6220 | noc/noc#1535 Change DIG - > DNSPython |
| MR6229 | Add Windows compatible which command. |
| MR6236 | Update monitoring-noc, add telegraf section |
| MR6252 | Catch ValueError on objectmetrics NBI API. |
| MR6263 | Add stage for upload binary dnssync |
| MR6266 | Remove always master docs build |
| MR6267 | Not setup signal.SIGHUP hanler for NT system. |
| MR6271 | noc/noc#1825 Add ManagedObject Query Source to GrafanaDS. |
| MR6274 | noc/noc#1821 Add managed_object to fm.ActiveEvent index for UI Filter. |
| MR6276 | Add SLAProbe Clickhouse Dictionary. |
| MR6282 | noc/noc#1825 Better variable configuration on Grafana JSON Datasource. |
| MR6286 | noc/noc#1831 Add iter_related_changed method for getting related model. |
| MR6302 | noc/noc#1831 datastream: get_changed_instance helper method |
| MR6324 | Add ensure_labels to ETL loader. |
| MR6346 | Add checkpoint param to etl extract command. |
| MR6347 | Cleanup Maintenance.direct_object when ManagedObject delete. |
| MR6360 | Exclude Auto generate fields from managedobject save. |
| MR6379 | Add task_stale_count metric to selfmon. |
| MR6388 | noc/noc#1860 Add forget-history to gridvcs command. |
| MR6390 | noc/noc#1837 Add old and new field value to change_fields structure. |
| MR6394 | noc/noc#1445 Add Label header to MX messages. |
| MR6400 | noc/noc#1851 Convert MessageRoute Settings to new format. |
| MR6426 | noc/noc#1846 Add description to ResourceGroup Search field. |
| MR6427 | noc/noc#1866 Add filterProtected attribute to UI labelfield fir Filter fields. |
| MR6434 | noc/noc#1875 Add noc alarms command for Alarms management. |
| MR6441 | Implement asyncio UDP-server |
| MR6459 | noc/noc#1869 Add UUIDParameter type to config. |
| MR6463 | Make CI upload docs |
| MR6465 | Update Brocade.IronWire profile |
| MR6470 | noc/noc#1369 Request user_info over bind_user for OpenLDAP Auth settings. |
| MR6479 | noc/noc#1884 Add row_class to Assigned Address API. |
| MR6491 | noc/noc#1851 Add cfgmxroute datastream for MX Services. |
| MR6502 | Add TTL support for django models in workflow decorator |
| MR6504 | Add connect_timeout to Stream.connect method. |
| MR6505 | Update Yandex Geocoder |
| MR6510 | noc/noc#1847 Add interface_status_change message type. |
| MR6511 | Extend use Scope param on Capability Item |
| MR6512 | Strip address value on ManagedObject |
| MR6524 | Huawei.MA5600T. Add Technology labels. |
| MR6531 | noc/noc#1889 |
| MR6543 | Add test for UDP-server |
| MR6546 | noc/noc#1816 Add escalation options on Maintenance |
| MR6548 | docs: Improve docker compose guide language and readability |
| MR6550 | Extend ip.Address model by status TTL fields |
| MR6552 | noc/noc#1710 Add lazy init Probe nodes on metrics service. |
| MR6557 | noc/noc#1895 Add load rules to Metrics service. |
| MR6558 | noc/noc#1755 Add param config to MetricAction model. |
| MR6559 | noc/noc#1891 Add DefaultRoute to MX. |
| MR6564 | Remove MX routes for default route. |
| MR6566 | noc/noc#1897 Add streaming call to metrics discovery. |
| MR6567 | Clean cli stream if not setup connection. |
| MR6578 | Speedup docs build. |
| MR6582 | Normalize capabilities value on report datasource. |
| MR6592 | Add cfgmetricsources datastream for Source Config on Metrics Service. |
| MR6593 | Improve cdag command for use clickhouse source as input. |
| MR6599 | Raise SNMPError BadValue exception when device return empty varbinds. |
| MR6610 | noc/noc#1357 Add MagicValue for set None metric value on Clickhouse. |
| MR6611 | noc/noc#1902 Remove ESMRE library from requirements |
| MR6613 | noc/noc#1851 Migrate ObjectNotification to MessageRoute. |
| MR6615 | Add Language Switcher to docs page. |
| MR6617 | Switch web-service from Tornado to FastAPI |
| MR6618 | Add composed metric to MetricType. |
| MR6619 | Change Dev docs structure to topic/type. |
| MR6623 | Add TimeZone param typo to GlobalConfig. |
| MR6626 | Add How-To NOC translation to docs. |
| MR6628 | Add How-to create object model. |
| MR6632 | noc/noc#1825 Add GrafanaDS docs. |
| MR6633 | Add workflow to interfaces |
| MR6644 | Add InterfaceProfile match label. |
| MR6645 | Add Interface weight to calculate alarm severity. |
| MR6646 | Add namespace to action nodes, for different Alarm rules. |
| MR6651 | noc/noc#1912 Reformat INV/Interface Application. |
| MR6653 | Move rust build from common jobs to native ci file |
| MR6654 | noc/noc#1899 Add required_rules to Workflow Transition. |
| MR6655 | Assign handling loop link error in NetworkSegment model to @tree decorator |
| MR6656 | Send state signal on interface status discovery. |
| MR6671 | #1918 CDAG node activation speedup |
| MR6681 | Move compose nodes to probes field. |
| MR6682 | Use lz4 for speedup metrics changelog. |
| MR6683 | Add proxy_metric param for send collected get_metrics from Discovery process. |
| MR6685 | Add arg0 to DiagnosticData. |
| MR6686 | Cleanup flake. |
| MR6688 | Catch SSHAuth error for processed on discovery side. |
| MR6690 | Update Mellanox.Onyx profile |
| MR6692 | Simplify credential diagnostic. |
| MR6694 | noc/noc#1783 Add metric_collected_policy setting to InterfaceProfile |
| MR6703 | Update JSONRPCAPI class |
| MR6704 | Skip HTTP proto for CLI diagnostic check. |
| MR6708 | Improve alarm command for Correlator proto use. |
| MR6712 | Speedup find cards by SourceConfig. |
| MR6716 | noc/noc#1930 Migrate Activator to FastAPI. |
| MR6722 | Bump requirements. |
| MR6725 | Speedup SLA probe UI. |
| MR6726 | Add hints to profile interface script. |
| MR6729 | Add part_no field to ManagedObject Stream to Asset section. |
| MR6730 | Add time_delta calc to metric service. |
| MR6741 | Fix HP.1905.get_mac_address_table script |
Bugfixes¶
| MR | Title |
|---|---|
| MR6247 | noc/noc#1821 Fix paging on Events UI. |
| MR6248 | Check TT on already_escalation method for fix Key Error when check notification doc. |
| MR6253 | Fix getting service on clear_alarm API view. |
| MR6265 | Fix filter server on dnszone DataStream. |
| MR6268 | Fix dictionary name in BI dashboard datasources. |
| MR6272 | Add migrate is_regex to is_matching field in Label. |
| MR6278 | TGSender fix markdown |
| MR6281 | Fix ETL extractors for incremental methods. |
| MR6295 | Fix Pydantic models for MRT Service request. |
| MR6299 | fix get_interfaces Alcatel.7302 |
| MR6304 | Fix Maintenance.update_affected_objects for None stop value. |
| MR6309 | Fix default format result on grafanads services query. |
| MR6312 | noc/noc#1451 Add ObjectId validator to ReferenceField. |
| MR6314 | Fix BI dict name on query to new dict name. |
| MR6326 | Add maintenance timepattern to get_active_maintenance. |
| MR6328 | Fix empty labels on etl ensure_labels. |
| MR6330 | Fix MultipleObjectsReturned on second resolve try. |
| MR6336 | Fix maintenance typo. |
| MR6341 | Fix Eltex.MES.get_inventory script |
| MR6366 | Fix MetricType on_delete Check for SLAProfile. |
| MR6373 | Delete FirmwarePolicy from ManagedObjectProfile delete check. |
| MR6380 | Catch Native TimeoutError exception on HTTP client request. |
| MR6395 | Fix Eltex.ESR.get_interfaces script |
| MR6402 | Fix error when hostname None value. |
| MR6412 | Fix DiscoveryID typo for None hostname. |
| MR6418 | Fix Grafanads Annotations |
| MR6423 | Fix ReportAlarmDetail - remove container_address |
| MR6425 | noc/noc#1872 Check vlanprofiles if create when vc migrate. |
| MR6431 | noc/noc#1858 Split table and view migrate on migrate-ch. |
| MR6439 | Fix clickhouse view schema migration |
| MR6447 | Add platform to FirmwarePolicy index |
| MR6448 | Add GIS set bi_id field migration. |
| MR6456 | Fix change MeasurementUnit on MetricType UI form. |
| MR6460 | Fix typo on MessageRoute migrate. |
| MR6466 | noc/noc#1876 Skip alarm with Disabled Escalation on always_first policies. |
| MR6469 | noc/noc!6469 Fix merge_fields function for bulk_change feature. |
| MR6483 | Fix decode_trap with raw_value. |
| MR6486 | Fix Event message generated for MX. |
| MR6490 | Fix HP.ProCurve profile |
| MR6515 | Fix typo on SAE. |
| MR6520 | Fix Dlink.DxS_Industrial_CLI.get_interfaces script |
| MR6536 | Fix change ip script |
| MR6538 | Fix Zyxel.MSAN.get_inventory script |
| MR6544 | Fix DLink.DxS_Industrial_CLI.get_lldp_neighbors script |
| MR6554 | Fix labelField in gridField |
| MR6555 | Fix in UDP-server test for asyncio UDP-server |
| MR6562 | noc/noc#1826 Fix calculate DNSZone changed when empty address FQDN. |
| MR6577 | Fix Workflow item saving |
| MR6579 | Fix SSH stream is None when close session. |
| MR6585 | Fix workflow migrate command |
| MR6586 | noc/noc#1900 Fix block datastream process client if no record in datastream. |
| MR6594 | Fix cdag expdecay Convert NS timestamp format fixed 'OverflowError: math range error'. |
| MR6597 | Fix error when empty varbinds on NO_SUCH_NAME SNMP error. |
| MR6612 | Fix Cisco.NXOS.get_lldp_neighbors script |
| MR6614 | fixing nav docs |
| MR6620 | Fix TagField for multiple value. |
| MR6622 | Fix suggest auth_profile migration to local type. |
| MR6627 | noc/noc#1909 Fix Raise Link Down alarm on interfacestatus discovery. |
| MR6639 | Noc fix l2domain profile #1888 |
| MR6641 | Fix compact state on Metrics service. |
| MR6660 | Fix DLink.DxS_Industrial_CLI.get_lldp_neighbors script |
| MR6663 | Fix detect UTC offset for metriccollector. |
| MR6666 | Fix ObjectNotification to MX Route Migration. |
| MR6677 | noc/noc#1925 Fix ReportPendingLink when empty DiscoveryID collection. |
| MR6689 | Fix typo on alarm name on metrics service. |
| MR6707 | Fix typo on SNMP protocol checker. |
| MR6709 | Set reopens when create Alarm. |
| MR6718 | Fix display ManagedObject diagnostic for unmanaged. |
| MR6721 | Fix type in CredentialChecker Raise privileges. |
| MR6723 | Fix getting user typo on jsonrpc API. |
| MR6724 | Fix check CredentialCheckRule for suggest_protocols. |
| MR6727 | Fix interface workflow state migration. |
| MR6731 | Fix check CredentialCheckRule for suggest_protocols. |
| MR6732 | Add scope param to get_by_metric_field. |
| MR6737 | Fix ManagedObject Attributes migrate caps. |
| MR6739 | Fix ManagedObject Attributes to CAPS migration. |
| MR6740 | Fix ManagedObject datastream typo. |
Code Cleanup¶
| MR | Title |
|---|---|
| MR6285 | noc/noc#1833 Force convert segment_path and container_path element to string. |
| MR6306 | Docker image: black 22.3.0, mypy 0.950 |
| MR6308 | Fix DNSZone serial field to persist. |
| MR6317 | Add type annotations to topology class. |
| MR6331 | Migrate match_version on scripts to matchers. |
| MR6348 | Type annotations for ConfDB DEF |
| MR6391 | Cleanup ManagedObject.data field use. |
| MR6473 | Convert bytes param value for consul params. |
| MR6526 | noc/noc#1886 Add str attribute to VLANTemplate for lookup name. |
| MR6576 | noc/noc#1857 Migrate Suggest AuthProfile to Credential Rules. |
| MR6648 | Add units to SLA metrics. |
| MR6687 | Add units to Profile Metrics. |
| MR6698 | Format change-ip script. |
| MR6705 | Bump ManagedObject cache version. |
| MR6713 | ./noc beef: Convert to f-strings |
| MR6714 | Cleanup unusable init file |
Profile Changes¶
Alcatel.7302¶
| MR | Title |
|---|---|
| MR6528 | Alcatel.7302. Add Operation error catch when CLI problem. |
| MR6551 | Alcatel.7302.get_interfaces. Fix interface labels typo. |
Cisco.IOS¶
| MR | Title |
|---|---|
| MR6396 | Fix Cisco.IOS.get_inventory script |
| MR6487 | noc/noc#1867 Cisco.IOS. Fix compare version regex for 16.6.5 format. |
DLink.DxS¶
| MR | Title |
|---|---|
| MR6199 | Fix DLink.DxS.get_mac_address_table script |
| MR6477 | DLink.DxS.get_inventory. Set stack number. |
Eltex.MES¶
| MR | Title |
|---|---|
| MR6364 | add new platform MES3348 Eltex.MES |
| MR6475 | Eltex.MES.get_inventory. Add number to stack device. |
Eltex.MES24xx¶
| MR | Title |
|---|---|
| MR6226 | Fix Eltex.MES24x.get_interfaces script |
| MR6484 | Eltex.MES24xx.get_inventory. Add tengigabitethernet for transceiver command. |
Generic¶
| MR | Title |
|---|---|
| MR6401 | Generic.get_metrics. Add units (Measurement Units) param to set_metrics. |
| MR6406 | Generic.get_metrics. Check SNMP for S access preference. |
| MR6410 | Generic.get_inventory. Add get_chassis_sensors method for return chassis sensor. |
| MR6583 | Generic.get_sla_probes based on DISMAN-PING-MIB example. |
| MR6587 | Generic.get_interfaces. Add method check a subinterface |
Hikvision.DSKV8¶
| MR | Title |
|---|---|
| MR6631 | Hikvision.DSKV8. Add http digest auth. |
Huawei.MA5600T¶
| MR | Title |
|---|---|
| MR6334 | Huawei.MA5600T.get_metrics. Fix labels format. |
| MR6570 | Huawei.MA5600T.get_inventory. Fix typo for detect sensor OID. |
| MR6624 | Add support H561SHEA board in Huawei.MA5600T.get_interfaces script |
| MR6696 | fix type ethernet interfaces Huawei MA58xx |
Huawei.VRP¶
| MR | Title |
|---|---|
| MR6245 | Huawei.VRP.get_version - delete \r in version |
| MR6476 | Huawei.VRP.get_portchannel. Add cleanup interface name from output. |
Juniper.JUNOS¶
| MR | Title |
|---|---|
| MR6581 | noc/noc#1898 Juniper.JUNOS.get_interfaces. Fix untagged_vlan typo. |
MikroTik.RouterOS¶
| MR | Title |
|---|---|
| MR6717 | Fix MikroTik.RouterOS.get_lldp_neighbors.script |
NAG.SNR¶
| MR | Title |
|---|---|
| MR6634 | Fix prompt match in NAG.SNR profile |
NSN.TIMOS¶
| MR | Title |
|---|---|
| MR6254 | NCN.TIMOS.get_metrics. Fix DHCP Metrics Used. |
| MR6386 | Fix NSN.TIMOS.get_lldp_neighbors script |
Qtech.QSW2800¶
| MR | Title |
|---|---|
| MR6464 | Fix Qtech.QSW2800.get_version script |
| MR6518 | add model QSW2800.get_version |
Raisecom.ROS¶
| MR | Title |
|---|---|
| MR6445 | fix Raisecom.ROS profile |
| MR6665 | Raisecom.ROS.get_version. Add parse_kv_version method for parse version output. |
Zyxel.MSAN¶
| MR | Title |
|---|---|
| MR6601 | Zyxel.MSAN.get_interfaces. Fix slot count in range. |
rare¶
| MR | Title |
|---|---|
| MR6246 | Meinberg.LANTIME. Cleanup black. |
| MR6344 | Add initial support for 3Com Superstack 3 5500 |
| MR6351 | fix "configure terminal" promt Eltex.MA4000 |
| MR6414 | Qtech.BFC_PBIC_S.get_inventory. Add sensor labels. |
| MR6416 | Ttronics.KUB. Add sensor labels. |
| MR6493 | Angtel.Topaz. Add SNMP support. |
| MR6497 | model fix in get_version Huawei.MA5300 |
| MR6532 | Add initial support for Mellanox.Cumulus profile |
| MR6568 | DLink.DAS.get_version. Fix variable 'r' referenced before assignment. |
| MR6574 | Linksys.SWR.login. Not used message for NOCError. |
| MR6580 | Juniper.EX2500. Fix flake. |
| MR6600 | ZTE.ZXDSL98xx.get_inventory. Fix fill_ports method use from profile. |
| MR6604 | NSCComm.LPOS. Add SNMP support to scripts. |
| MR6711 | Rotek.RTBS. Fix metric units. |
Collections Changes¶
| MR | Title |
|---|---|
| MR6361 | Add DOM Errors metric. |
| MR6384 | Update NAG.SNR profile |
| MR6435 | add profilecheckrule SecurityCode.Kontinent |
| MR6478 | Remove disposition from 'Security | Authentication | SNMP Authentication Failure' rule. |
| MR6489 | Add Eltex.SMG profilecheckrule |
| MR6517 | add qtech profilecheckrule |
| MR6542 | noc/noc#534 Add DiagnosticCheck to Discovery. |
| MR6553 | Fix whitespace in path |
| MR6591 | Add Zyxel.MSAN profilecheckrule |
| MR6635 | New metrics |
| MR6636 | Add compose expression to MetricAction. |
| MR6649 | noc/noc#1915 Migrate ManagedObject Attributes to Capabilities. |
| MR6661 | Speedup create probe on metrics service. |
| MR6684 | Cleanup CDAG subgraph implementation |
| MR6693 | add model huawei MA5800-X2 |
| MR6706 | Add units to 'CPU | Usage' metric |
| MR6710 | Replace vcds alarm datasource to vlands. |
| MR6715 | update models QSW-8200-28F-AC-DC, QSW-8200-28F-DC-DC |
Deploy Changes¶
| MR | Title |
|---|---|
| MR6241 | Ansible: fix deprecation warning |
| MR6291 | Migrate to Gufo Ping |
| MR6298 | Ansible fix pg init command |
| MR6301 | change repo clickhouse |
| MR6342 | Ansible fix version string type |
| MR6359 | Ansible fix apt key proxy problem |
| MR6368 | Remove ch_datasource service from deploy |
| MR6372 | add cheking certificates for nats and liftbridge |
| MR6403 | Fix deploy consul svc-set |
| MR6436 | fix tower_run_checks |
| MR6492 | Add supporting ubuntu22 |
| MR6537 | Fix kafkasender checks condition |
| MR6540 | change repo for libssl1.1 |
| MR6675 | fix special character in postgres password |
| MR6678 | Rewrite noc_python part of noc role |
| MR6691 | Ansible drop debian9 deploy support |
| MR6700 | change plugin SimpleJson on simpod-json-datasource |
| MR6702 | Add backup slots for metrics |
| MR6738 | Fix indent in patch |