NOC 22.2

14.10.2022

NOC 22.2

22.2 release contains 286 bugfixes, optimisations and improvements.

Highlights

Gufo Labs Collaboration

Gufo Labs is a private IT-consulting company based in Milan, Italy. Among other things, they focus on extracting the vital parts of the NOC technologies, reworking them, and releasing them as separate open-source packages. Separation into independent packages allows a focus on API, documentation, reliability, portability, resource usage, and performance.

Known as Gufo Stack, this software is the valuable tool not only for NOC but every network-related engineer and developer.

ETL

ETL system got the ability to fetch data incrementally. Depending on the remote system, ETL can request only changed data. It reduces imposed load and allows reducing of the synchronization intervals.

Additionally, ETL now can extract and load Managed Object’s capabilities.

MX

MX service got the ability to route and process messages based on labels. To reduce the system resource usage, parts of mx service can be integrated directly in the producers’ processes.

The Fault Management got the ability to mirror the messages to the mx, allowing real-time export to the Kafka. syslogcollector can mirror incoming syslog messages, trapcollector can mirror semi-parsed SNMP Trap PDUs, while classifier can mirror the classified messages. This greatly increases the NOC’s capability to pump the Data Lakes and all the other’s “Big Data” stuff.

Metrics service

After several profiling sessions, the metrics service requires much less CPU and memory.

Ping

NOC has switched to the Gufo Ping. Gufo Ping is the successor of the NOC’s Cython-accelerated ping implementation. Compared to the previous implementation it has distinctive features:

  • Implemented in Rust language using pyo3 bindings.
  • Memory and CPU optimized.
  • 100% test coverage.
  • Built-in benchmarks.
  • Reduced context switches.
  • Reduced python-to-binary switches.
  • Applies BPF filter to separate concurrent ping processes.
  • Packaged as binary wheels for the major platforms.

Python

Python 3.7 support is deprecated, leaving support for 3.8-3.10.

Internals

  • tornado has been removed from all critical paths. Next version will remove explicit tornado requirements.

New features

MR Title
!6264 noc/noc#1780 Incremental ETL
!6422 noc/noc#1861 Add send snmptrap message to mx.
!6433 noc/noc#1836 Add StreamingConfig param to Activator.script RPC call.
!6437 noc/noc#1351 Add capabilities to ManagedObject ETL extractor.
!6442 noc/noc#1861 Add send syslog message to mx.
!6451 noc/noc#1838 Add event to mx message.
!6506 noc/noc#1857 Add CredentialCheckRule model.
!6509 noc/noc#1853 Add diagnostics to ManagedObject.
!6530 noc/noc#534 Add Diagnostic discovery.
!6556 noc/noc#534 Add User-defined ObjectDiagnosticConfig
!6669 noc/noc#1907 Move Message Router to core.

Improvements

MR Title
!5084 noc/noc#1820 Add hostname_id to DiscoveryID model
!6211 Implement message storm protection
!6220 noc/noc#1535 Change DIG - > DNSPython
!6229 Add Windows compatible which command.
!6236 Update monitoring-noc, add telegraf section
!6252 Catch ValueError on objectmetrics NBI API.
!6263 Add stage for upload binary dnssync
!6266 Remove always master docs build
!6267 Not setup signal.SIGHUP hanler for NT system.
!6271 noc/noc#1825 Add ManagedObject Query Source to GrafanaDS.
!6274 noc/noc#1821 Add managed_object to fm.ActiveEvent index for UI Filter.
!6276 Add SLAProbe Clickhouse Dictionary.
!6282 noc/noc#1825 Better variable configuration on Grafana JSON Datasource.
!6286 noc/noc#1831 Add iter_related_changed method for getting related model.
!6302 noc/noc#1831 datastream: get_changed_instance helper method
!6324 Add ensure_labels to ETL loader.
!6346 Add checkpoint param to etl extract command.
!6347 Cleanup Maintenance.direct_object when ManagedObject delete.
!6360 Exclude Auto generate fields from managedobject save.
!6379 Add task_stale_count metric to selfmon.
!6388 noc/noc#1860 Add forget-history to gridvcs command.
!6390 noc/noc#1837 Add old and new field value to change_fields structure.
!6394 noc/noc#1445 Add Label header to MX messages.
!6400 noc/noc#1851 Convert MessageRoute Settings to new format.
!6426 noc/noc#1846 Add description to ResourceGroup Search field.
!6427 noc/noc#1866 Add filterProtected attribute to UI labelfield fir Filter fields.
!6434 noc/noc#1875 Add noc alarms command for Alarms management.
!6441 Implement asyncio UDP-server
!6459 noc/noc#1869 Add UUIDParameter type to config.
!6463 Make CI upload docs
!6465 Update Brocade.IronWire profile
!6470 noc/noc#1369 Request user_info over bind_user for OpenLDAP Auth settings.
!6479 noc/noc#1884 Add row_class to Assigned Address API.
!6491 noc/noc#1851 Add cfgmxroute datastream for MX Services.
!6502 Add TTL support for django models in workflow decorator
!6504 Add connect_timeout to Stream.connect method.
!6505 Update Yandex Geocoder
!6510 noc/noc#1847 Add interface_status_change message type.
!6511 Extend use Scope param on Capability Item
!6512 Strip address value on ManagedObject
!6524 Huawei.MA5600T. Add Technology labels.
!6531 noc/noc#1889
!6543 Add test for UDP-server
!6546 noc/noc#1816 Add escalation options on Maintenance
!6548 docs: Improve docker compose guide language and readability
!6550 Extend ip.Address model by status TTL fields
!6552 noc/noc#1710 Add lazy init Probe nodes on metrics service.
!6557 noc/noc#1895 Add load rules to Metrics service.
!6558 noc/noc#1755 Add param config to MetricAction model.
!6559 noc/noc#1891 Add DefaultRoute to MX.
!6564 Remove MX routes for default route.
!6566 noc/noc#1897 Add streaming call to metrics discovery.
!6567 Clean cli stream if not setup connection.
!6578 Speedup docs build.
!6582 Normalize capabilities value on report datasource.
!6592 Add cfgmetricsources datastream for Source Config on Metrics Service.
!6593 Improve cdag command for use clickhouse source as input.
!6599 Raise SNMPError BadValue exception when device return empty varbinds.
!6610 noc/noc#1357 Add MagicValue for set None metric value on Clickhouse.
!6611 noc/noc#1902 Remove ESMRE library from requirements
!6613 noc/noc#1851 Migrate ObjectNotification to MessageRoute.
!6615 Add Language Switcher to docs page.
!6617 Switch web-service from Tornado to FastAPI
!6618 Add composed metric to MetricType.
!6619 Change Dev docs structure to topic/type.
!6623 Add TimeZone param typo to GlobalConfig.
!6626 Add How-To NOC translation to docs.
!6628 Add How-to create object model.
!6632 noc/noc#1825 Add GrafanaDS docs.
!6633 Add workflow to interfaces
!6644 Add InterfaceProfile match label.
!6645 Add Interface weight to calculate alarm severity.
!6646 Add namespace to action nodes, for different Alarm rules.
!6651 noc/noc#1912 Reformat INV/Interface Application.
!6653 Move rust build from common jobs to native ci file
!6654 noc/noc#1899 Add required_rules to Workflow Transition.
!6655 Assign handling loop link error in NetworkSegment model to @tree decorator
!6656 Send state signal on interface status discovery.
!6671 #1918 CDAG node activation speedup
!6681 Move compose nodes to probes field.
!6682 Use lz4 for speedup metrics changelog.
!6683 Add proxy_metric param for send collected get_metrics from Discovery process.
!6685 Add arg0 to DiagnosticData.
!6686 Cleanup flake.
!6688 Catch SSHAuth error for processed on discovery side.
!6690 Update Mellanox.Onyx profile
!6692 Simplify credential diagnostic.
!6694 noc/noc#1783 Add metric_collected_policy setting to InterfaceProfile
!6703 Update JSONRPCAPI class
!6704 Skip HTTP proto for CLI diagnostic check.
!6708 Improve alarm command for Correlator proto use.
!6712 Speedup find cards by SourceConfig.
!6716 noc/noc#1930 Migrate Activator to FastAPI.
!6722 Bump requirements.
!6725 Speedup SLA probe UI.
!6726 Add hints to profile interface script.
!6729 Add part_no field to ManagedObject Stream to Asset section.
!6730 Add time_delta calc to metric service.
!6741 Fix HP.1905.get_mac_address_table script

Bugfixes

MR Title
!6247 noc/noc#1821 Fix paging on Events UI.
!6248 Check TT on already_escalation method for fix Key Error when check notification doc.
!6253 Fix getting service on clear_alarm API view.
!6265 Fix filter server on dnszone DataStream.
!6268 Fix dictionary name in BI dashboard datasources.
!6272 Add migrate is_regex to is_matching field in Label.
!6278 TGSender fix markdown
!6281 Fix ETL extractors for incremental methods.
!6295 Fix Pydantic models for MRT Service request.
!6299 fix get_interfaces Alcatel.7302
!6304 Fix Maintenance.update_affected_objects for None stop value.
!6309 Fix default format result on grafanads services query.
!6312 noc/noc#1451 Add ObjectId validator to ReferenceField.
!6314 Fix BI dict name on query to new dict name.
!6326 Add maintenance timepattern to get_active_maintenance.
!6328 Fix empty labels on etl ensure_labels.
!6330 Fix MultipleObjectsReturned on second resolve try.
!6336 Fix maintenance typo.
!6341 Fix Eltex.MES.get_inventory script
!6366 Fix MetricType on_delete Check for SLAProfile.
!6373 Delete FirmwarePolicy from ManagedObjectProfile delete check.
!6380 Catch Native TimeoutError exception on HTTP client request.
!6395 Fix Eltex.ESR.get_interfaces script
!6402 Fix error when hostname None value.
!6412 Fix DiscoveryID typo for None hostname.
!6418 Fix Grafanads Annotations
!6423 Fix ReportAlarmDetail - remove container_address
!6425 noc/noc#1872 Check vlanprofiles if create when vc migrate.
!6431 noc/noc#1858 Split table and view migrate on migrate-ch.
!6439 Fix clickhouse view schema migration
!6447 Add platform to FirmwarePolicy index
!6448 Add GIS set bi_id field migration.
!6456 Fix change MeasurementUnit on MetricType UI form.
!6460 Fix typo on MessageRoute migrate.
!6466 noc/noc#1876 Skip alarm with Disabled Escalation on always_first policies.
!6469 noc/noc!6469 Fix merge_fields function for bulk_change feature.
!6483 Fix decode_trap with raw_value.
!6486 Fix Event message generated for MX.
!6490 Fix HP.ProCurve profile
!6515 Fix typo on SAE.
!6520 Fix Dlink.DxS_Industrial_CLI.get_interfaces script
!6536 Fix change ip script
!6538 Fix Zyxel.MSAN.get_inventory script
!6544 Fix DLink.DxS_Industrial_CLI.get_lldp_neighbors script
!6554 Fix labelField in gridField
!6555 Fix in UDP-server test for asyncio UDP-server
!6562 noc/noc#1826 Fix calculate DNSZone changed when empty address FQDN.
!6577 Fix Workflow item saving
!6579 Fix SSH stream is None when close session.
!6585 Fix workflow migrate command
!6586 noc/noc#1900 Fix block datastream process client if no record in datastream.
!6594 Fix cdag expdecay Convert NS timestamp format fixed ‘OverflowError: math range error’.
!6597 Fix error when empty varbinds on NO_SUCH_NAME SNMP error.
!6612 Fix Cisco.NXOS.get_lldp_neighbors script
!6614 fixing nav docs
!6620 Fix TagField for multiple value.
!6622 Fix suggest auth_profile migration to local type.
!6627 noc/noc#1909 Fix Raise Link Down alarm on interfacestatus discovery.
!6639 Noc fix l2domain profile #1888
!6641 Fix compact state on Metrics service.
!6660 Fix DLink.DxS_Industrial_CLI.get_lldp_neighbors script
!6663 Fix detect UTC offset for metriccollector.
!6666 Fix ObjectNotification to MX Route Migration.
!6677 noc/noc#1925 Fix ReportPendingLink when empty DiscoveryID collection.
!6689 Fix typo on alarm name on metrics service.
!6707 Fix typo on SNMP protocol checker.
!6709 Set reopens when create Alarm.
!6718 Fix display ManagedObject diagnostic for unmanaged.
!6721 Fix type in CredentialChecker Raise privileges.
!6723 Fix getting user typo on jsonrpc API.
!6724 Fix check CredentialCheckRule for suggest_protocols.
!6727 Fix interface workflow state migration.
!6731 Fix check CredentialCheckRule for suggest_protocols.
!6732 Add scope param to get_by_metric_field.
!6737 Fix ManagedObject Attributes migrate caps.
!6739 Fix ManagedObject Attributes to CAPS migration.
!6740 Fix ManagedObject datastream typo.

Code Cleanup

MR Title
!6285 noc/noc#1833 Force convert segment_path and container_path element to string.
!6306 Docker image: black 22.3.0, mypy 0.950
!6308 Fix DNSZone serial field to persist.
!6317 Add type annotations to topology class.
!6331 Migrate match_version on scripts to matchers.
!6348 Type annotations for ConfDB DEF
!6391 Cleanup ManagedObject.data field use.
!6473 Convert bytes param value for consul params.
!6526 noc/noc#1886 Add str attribute to VLANTemplate for lookup name.
!6576 noc/noc#1857 Migrate Suggest AuthProfile to Credential Rules.
!6648 Add units to SLA metrics.
!6687 Add units to Profile Metrics.
!6698 Format change-ip script.
!6705 Bump ManagedObject cache version.
!6713 ./noc beef: Convert to f-strings
!6714 Cleanup unusable init file

Profile Changes

Alcatel.7302

MR Title
!6528 Alcatel.7302. Add Operation error catch when CLI problem.
!6551 Alcatel.7302.get_interfaces. Fix interface labels typo.

Cisco.IOS

MR Title
!6396 Fix Cisco.IOS.get_inventory script
!6487 noc/noc#1867 Cisco.IOS. Fix compare version regex for 16.6.5 format.
MR Title
!6199 Fix DLink.DxS.get_mac_address_table script
!6477 DLink.DxS.get_inventory. Set stack number.

Eltex.MES

MR Title
!6364 add new platform MES3348 Eltex.MES
!6475 Eltex.MES.get_inventory. Add number to stack device.

Eltex.MES24xx

MR Title
!6226 Fix Eltex.MES24x.get_interfaces script
!6484 Eltex.MES24xx.get_inventory. Add tengigabitethernet for transceiver command.

Generic

MR Title
!6401 Generic.get_metrics. Add units (Measurement Units) param to set_metrics.
!6406 Generic.get_metrics. Check SNMP for S access preference.
!6410 Generic.get_inventory. Add get_chassis_sensors method for return chassis sensor.
!6583 Generic.get_sla_probes based on DISMAN-PING-MIB example.
!6587 Generic.get_interfaces. Add method check a subinterface

Hikvision.DSKV8

MR Title
!6631 Hikvision.DSKV8. Add http digest auth.

Huawei.MA5600T

MR Title
!6334 Huawei.MA5600T.get_metrics. Fix labels format.
!6570 Huawei.MA5600T.get_inventory. Fix typo for detect sensor OID.
!6624 Add support H561SHEA board in Huawei.MA5600T.get_interfaces script
!6696 fix type ethernet interfaces Huawei MA58xx

Huawei.VRP

MR Title
!6245 Huawei.VRP.get_version - delete \r in version
!6476 Huawei.VRP.get_portchannel. Add cleanup interface name from output.

Juniper.JUNOS

MR Title
!6581 noc/noc#1898 Juniper.JUNOS.get_interfaces. Fix untagged_vlan typo.

MikroTik.RouterOS

MR Title
!6717 Fix MikroTik.RouterOS.get_lldp_neighbors.script

NAG.SNR

MR Title
!6634 Fix prompt match in NAG.SNR profile

NSN.TIMOS

MR Title
!6254 NCN.TIMOS.get_metrics. Fix DHCP Metrics Used.
!6386 Fix NSN.TIMOS.get_lldp_neighbors script

Qtech.QSW2800

MR Title
!6464 Fix Qtech.QSW2800.get_version script
!6518 add model QSW2800.get_version

Raisecom.ROS

MR Title
!6445 fix Raisecom.ROS profile
!6665 Raisecom.ROS.get_version. Add parse_kv_version method for parse version output.

Zyxel.MSAN

MR Title
!6601 Zyxel.MSAN.get_interfaces. Fix slot count in range.

rare

MR Title
!6246 Meinberg.LANTIME. Cleanup black.
!6344 Add initial support for 3Com Superstack 3 5500
!6351 fix “configure terminal” promt Eltex.MA4000
!6414 Qtech.BFC_PBIC_S.get_inventory. Add sensor labels.
!6416 Ttronics.KUB. Add sensor labels.
!6493 Angtel.Topaz. Add SNMP support.
!6497 model fix in get_version Huawei.MA5300
!6532 Add initial support for Mellanox.Cumulus profile
!6568 DLink.DAS.get_version. Fix variable ‘r’ referenced before assignment.
!6574 Linksys.SWR.login. Not used message for NOCError.
!6580 Juniper.EX2500. Fix flake.
!6600 ZTE.ZXDSL98xx.get_inventory. Fix fill_ports method use from profile.
!6604 NSCComm.LPOS. Add SNMP support to scripts.
!6711 Rotek.RTBS. Fix metric units.

Collections Changes

MR Title
!6361 Add DOM Errors metric.
!6384 Update NAG.SNR profile
!6435 add profilecheckrule SecurityCode.Kontinent
!6478 Remove disposition from ‘Security | Authentication | SNMP Authentication Failure’ rule.
!6489 Add Eltex.SMG profilecheckrule
!6517 add qtech profilecheckrule
!6542 noc/noc#534 Add DiagnosticCheck to Discovery.
!6553 Fix whitespace in path
!6591 Add Zyxel.MSAN profilecheckrule
!6635 New metrics
!6636 Add compose expression to MetricAction.
!6649 noc/noc#1915 Migrate ManagedObject Attributes to Capabilities.
!6661 Speedup create probe on metrics service.
!6684 Cleanup CDAG subgraph implementation
!6693 add model huawei MA5800-X2
!6706 Add units to ‘CPU | Usage’ metric
!6710 Replace vcds alarm datasource to vlands.
!6715 update models QSW-8200-28F-AC-DC, QSW-8200-28F-DC-DC

Deploy Changes

MR Title
!6241 Ansible: fix deprecation warning
!6291 Migrate to Gufo Ping
!6298 Ansible fix pg init command
!6301 change repo clickhouse
!6342 Ansible fix version string type
!6359 Ansible fix apt key proxy problem
!6368 Remove ch_datasource service from deploy
!6372 add cheking certificates for nats and liftbridge
!6403 Fix deploy consul svc-set
!6436 fix tower_run_checks
!6492 Add supporting ubuntu22
!6537 Fix kafkasender checks condition
!6540 change repo for libssl1.1
!6675 fix special character in postgres password
!6678 Rewrite noc_python part of noc role
!6691 Ansible drop debian9 deploy support
!6700 change plugin SimpleJson on simpod-json-datasource
!6702 Add backup slots for metrics
!6738 Fix indent in patch

Search

Categories

Tags