Перейти к содержанию

NOC 22.2

22.2 release contains 286 bugfixes, optimisations and improvements.

Highlights

Gufo Labs Collaboration

Gufo Labs is a private IT-consulting company based in Milan, Italy. Among other things, they focus on extracting the vital parts of the NOC technologies, reworking them, and releasing them as separate open-source packages. Separation into independent packages allows a focus on API, documentation, reliability, portability, resource usage, and performance.

Known as Gufo Stack, this software is the valuable tool not only for NOC but every network-related engineer and developer.

ETL

ETL system got the ability to fetch data incrementally. Depending on the remote system, ETL can request only changed data. It reduces imposed load and allows reducing of the synchronization intervals.

Additionally, ETL now can extract and load Managed Object's capabilities.

MX

MX service got the ability to route and process messages based on labels. To reduce the system resoure usage, parts of mx service can be integrated directly in the producers' processes.

The Fault Management got the ability to mirror the messages to the mx, allowing real-time export to the Kafka. syslogcollector can mirror incoming syslog messages, trapcollector can mirror semi-parsed SNMP Trap PDUs, while classifier can mirror the classified messages. This greatly increases the NOC's capability to pump the Data Lakes and all the other's "Big Data" stuff.

Metrics service

After several profiling sessions, the metrics service requires much less CPU and memory.

Ping

NOC has switched to the Gufo Ping. Gufo Ping is the successor of the NOC's Cython-accelerated ping implementation. Compared to the previous implementation it has distinctive features:

  • Implemented in Rust language using pyo3 bindings.
  • Memory and CPU optimized.
  • 100% test coverage.
  • Built-in benchmarks.
  • Reduced context switches.
  • Reduced python-to-binary switches.
  • Applies BPF filter to separate concurrent ping processes.
  • Packaged as binary wheels for the major platforms.

Python

Python 3.7 support is deprecated, leaving support for 3.8-3.10.

Internals

  • tornado has been removed from all critical paths. Next version will remove explicit tornado requirements.

New features

MRTitle
MR6264noc/noc#1780 Incremental ETL
MR6422noc/noc#1861 Add send snmptrap message to mx.
MR6433noc/noc#1836 Add StreamingConfig param to Activator.script RPC call.
MR6437noc/noc#1351 Add capabilities to ManagedObject ETL extractor.
MR6442noc/noc#1861 Add send syslog message to mx.
MR6451noc/noc#1838 Add event to mx message.
MR6506noc/noc#1857 Add CredentialCheckRule model.
MR6509noc/noc#1853 Add diagnostics to ManagedObject.
MR6530noc/noc#534 Add Diagnostic discovery.
MR6556noc/noc#534 Add User-defined ObjectDiagnosticConfig
MR6669noc/noc#1907 Move Message Router to core.

Improvements

MRTitle
MR5084noc/noc#1820 Add hostname_id to DiscoveryID model
MR6211Implement message storm protection
MR6220noc/noc#1535 Change DIG - > DNSPython
MR6229Add Windows compatible which command.
MR6236Update monitoring-noc, add telegraf section
MR6252Catch ValueError on objectmetrics NBI API.
MR6263Add stage for upload binary dnssync
MR6266Remove always master docs build
MR6267Not setup signal.SIGHUP hanler for NT system.
MR6271noc/noc#1825 Add ManagedObject Query Source to GrafanaDS.
MR6274noc/noc#1821 Add managed_object to fm.ActiveEvent index for UI Filter.
MR6276Add SLAProbe Clickhouse Dictionary.
MR6282noc/noc#1825 Better variable configuration on Grafana JSON Datasource.
MR6286noc/noc#1831 Add iter_related_changed method for getting related model.
MR6302noc/noc#1831 datastream: get_changed_instance helper method
MR6324Add ensure_labels to ETL loader.
MR6346Add checkpoint param to etl extract command.
MR6347Cleanup Maintenance.direct_object when ManagedObject delete.
MR6360Exclude Auto generate fields from managedobject save.
MR6379Add task_stale_count metric to selfmon.
MR6388noc/noc#1860 Add forget-history to gridvcs command.
MR6390noc/noc#1837 Add old and new field value to change_fields structure.
MR6394noc/noc#1445 Add Label header to MX messages.
MR6400noc/noc#1851 Convert MessageRoute Settings to new format.
MR6426noc/noc#1846 Add description to ResourceGroup Search field.
MR6427noc/noc#1866 Add filterProtected attribute to UI labelfield fir Filter fields.
MR6434noc/noc#1875 Add noc alarms command for Alarms management.
MR6441Implement asyncio UDP-server
MR6459noc/noc#1869 Add UUIDParameter type to config.
MR6463Make CI upload docs
MR6465Update Brocade.IronWire profile
MR6470noc/noc#1369 Request user_info over bind_user for OpenLDAP Auth settings.
MR6479noc/noc#1884 Add row_class to Assigned Address API.
MR6491noc/noc#1851 Add cfgmxroute datastream for MX Services.
MR6502Add TTL support for django models in workflow decorator
MR6504Add connect_timeout to Stream.connect method.
MR6505Update Yandex Geocoder
MR6510noc/noc#1847 Add interface_status_change message type.
MR6511Extend use Scope param on Capability Item
MR6512Strip address value on ManagedObject
MR6524Huawei.MA5600T. Add Technology labels.
MR6531noc/noc#1889
MR6543Add test for UDP-server
MR6546noc/noc#1816 Add escalation options on Maintenance
MR6548docs: Improve docker compose guide language and readability
MR6550Extend ip.Address model by status TTL fields
MR6552noc/noc#1710 Add lazy init Probe nodes on metrics service.
MR6557noc/noc#1895 Add load rules to Metrics service.
MR6558noc/noc#1755 Add param config to MetricAction model.
MR6559noc/noc#1891 Add DefaultRoute to MX.
MR6564Remove MX routes for default route.
MR6566noc/noc#1897 Add streaming call to metrics discovery.
MR6567Clean cli stream if not setup connection.
MR6578Speedup docs build.
MR6582Normalize capabilities value on report datasource.
MR6592Add cfgmetricsources datastream for Source Config on Metrics Service.
MR6593Improve cdag command for use clickhouse source as input.
MR6599Raise SNMPError BadValue exception when device return empty varbinds.
MR6610noc/noc#1357 Add MagicValue for set None metric value on Clickhouse.
MR6611noc/noc#1902 Remove ESMRE library from requirements
MR6613noc/noc#1851 Migrate ObjectNotification to MessageRoute.
MR6615Add Language Switcher to docs page.
MR6617Switch web-service from Tornado to FastAPI
MR6618Add composed metric to MetricType.
MR6619Change Dev docs structure to topic/type.
MR6623Add TimeZone param typo to GlobalConfig.
MR6626Add How-To NOC translation to docs.
MR6628Add How-to create object model.
MR6632noc/noc#1825 Add GrafanaDS docs.
MR6633Add workflow to interfaces
MR6644Add InterfaceProfile match label.
MR6645Add Interface weight to calculate alarm severity.
MR6646Add namespace to action nodes, for different Alarm rules.
MR6651noc/noc#1912 Reformat INV/Interface Application.
MR6653Move rust build from common jobs to native ci file
MR6654noc/noc#1899 Add required_rules to Workflow Transition.
MR6655Assign handling loop link error in NetworkSegment model to @tree decorator
MR6656Send state signal on interface status discovery.
MR6671#1918 CDAG node activation speedup
MR6681Move compose nodes to probes field.
MR6682Use lz4 for speedup metrics changelog.
MR6683Add proxy_metric param for send collected get_metrics from Discovery process.
MR6685Add arg0 to DiagnosticData.
MR6686Cleanup flake.
MR6688Catch SSHAuth error for processed on discovery side.
MR6690Update Mellanox.Onyx profile
MR6692Simplify credential diagnostic.
MR6694noc/noc#1783 Add metric_collected_policy setting to InterfaceProfile
MR6703Update JSONRPCAPI class
MR6704Skip HTTP proto for CLI diagnostic check.
MR6708Improve alarm command for Correlator proto use.
MR6712Speedup find cards by SourceConfig.
MR6716noc/noc#1930 Migrate Activator to FastAPI.
MR6722Bump requirements.
MR6725Speedup SLA probe UI.
MR6726Add hints to profile interface script.
MR6729Add part_no field to ManagedObject Stream to Asset section.
MR6730Add time_delta calc to metric service.
MR6741Fix HP.1905.get_mac_address_table script

Bugfixes

MRTitle
MR6247noc/noc#1821 Fix paging on Events UI.
MR6248Check TT on already_escalation method for fix Key Error when check notification doc.
MR6253Fix getting service on clear_alarm API view.
MR6265Fix filter server on dnszone DataStream.
MR6268Fix dictionary name in BI dashboard datasources.
MR6272Add migrate is_regex to is_matching field in Label.
MR6278TGSender fix markdown
MR6281Fix ETL extractors for incremental methods.
MR6295Fix Pydantic models for MRT Service request.
MR6299fix get_interfaces Alcatel.7302
MR6304Fix Maintenance.update_affected_objects for None stop value.
MR6309Fix default format result on grafanads services query.
MR6312noc/noc#1451 Add ObjectId validator to ReferenceField.
MR6314Fix BI dict name on query to new dict name.
MR6326Add maintenance timepattern to get_active_maintenance.
MR6328Fix empty labels on etl ensure_labels.
MR6330Fix MultipleObjectsReturned on second resolve try.
MR6336Fix maintenance typo.
MR6341Fix Eltex.MES.get_inventory script
MR6366Fix MetricType on_delete Check for SLAProfile.
MR6373Delete FirmwarePolicy from ManagedObjectProfile delete check.
MR6380Catch Native TimeoutError exception on HTTP client request.
MR6395Fix Eltex.ESR.get_interfaces script
MR6402Fix error when hostname None value.
MR6412Fix DiscoveryID typo for None hostname.
MR6418Fix Grafanads Annotations
MR6423Fix ReportAlarmDetail - remove container_address
MR6425noc/noc#1872 Check vlanprofiles if create when vc migrate.
MR6431noc/noc#1858 Split table and view migrate on migrate-ch.
MR6439Fix clickhouse view schema migration
MR6447Add platform to FirmwarePolicy index
MR6448Add GIS set bi_id field migration.
MR6456Fix change MeasurementUnit on MetricType UI form.
MR6460Fix typo on MessageRoute migrate.
MR6466noc/noc#1876 Skip alarm with Disabled Escalation on always_first policies.
MR6469noc/noc!6469 Fix merge_fields function for bulk_change feature.
MR6483Fix decode_trap with raw_value.
MR6486Fix Event message generated for MX.
MR6490Fix HP.ProCurve profile
MR6515Fix typo on SAE.
MR6520Fix Dlink.DxS_Industrial_CLI.get_interfaces script
MR6536Fix change ip script
MR6538Fix Zyxel.MSAN.get_inventory script
MR6544Fix DLink.DxS_Industrial_CLI.get_lldp_neighbors script
MR6554Fix labelField in gridField
MR6555Fix in UDP-server test for asyncio UDP-server
MR6562noc/noc#1826 Fix calculate DNSZone changed when empty address FQDN.
MR6577Fix Workflow item saving
MR6579Fix SSH stream is None when close session.
MR6585Fix workflow migrate command
MR6586noc/noc#1900 Fix block datastream process client if no record in datastream.
MR6594Fix cdag expdecay Convert NS timestamp format fixed 'OverflowError: math range error'.
MR6597Fix error when empty varbinds on NO_SUCH_NAME SNMP error.
MR6612Fix Cisco.NXOS.get_lldp_neighbors script
MR6614fixing nav docs
MR6620Fix TagField for multiple value.
MR6622Fix suggest auth_profile migration to local type.
MR6627noc/noc#1909 Fix Raise Link Down alarm on interfacestatus discovery.
MR6639Noc fix l2domain profile #1888
MR6641Fix compact state on Metrics service.
MR6660Fix DLink.DxS_Industrial_CLI.get_lldp_neighbors script
MR6663Fix detect UTC offset for metriccollector.
MR6666Fix ObjectNotification to MX Route Migration.
MR6677noc/noc#1925 Fix ReportPendingLink when empty DiscoveryID collection.
MR6689Fix typo on alarm name on metrics service.
MR6707Fix typo on SNMP protocol checker.
MR6709Set reopens when create Alarm.
MR6718Fix display ManagedObject diagnostic for unmanaged.
MR6721Fix type in CredentialChecker Raise privileges.
MR6723Fix getting user typo on jsonrpc API.
MR6724Fix check CredentialCheckRule for suggest_protocols.
MR6727Fix interface workflow state migration.
MR6731Fix check CredentialCheckRule for suggest_protocols.
MR6732Add scope param to get_by_metric_field.
MR6737Fix ManagedObject Attributes migrate caps.
MR6739Fix ManagedObject Attributes to CAPS migration.
MR6740Fix ManagedObject datastream typo.

Code Cleanup

MRTitle
MR6285noc/noc#1833 Force convert segment_path and container_path element to string.
MR6306Docker image: black 22.3.0, mypy 0.950
MR6308Fix DNSZone serial field to persist.
MR6317Add type annotations to topology class.
MR6331Migrate match_version on scripts to matchers.
MR6348Type annotations for ConfDB DEF
MR6391Cleanup ManagedObject.data field use.
MR6473Convert bytes param value for consul params.
MR6526noc/noc#1886 Add str attribute to VLANTemplate for lookup name.
MR6576noc/noc#1857 Migrate Suggest AuthProfile to Credential Rules.
MR6648Add units to SLA metrics.
MR6687Add units to Profile Metrics.
MR6698Format change-ip script.
MR6705Bump ManagedObject cache version.
MR6713./noc beef: Convert to f-strings
MR6714Cleanup unusable init file

Profile Changes

Alcatel.7302

MRTitle
MR6528Alcatel.7302. Add Operation error catch when CLI problem.
MR6551Alcatel.7302.get_interfaces. Fix interface labels typo.

Cisco.IOS

MRTitle
MR6396Fix Cisco.IOS.get_inventory script
MR6487noc/noc#1867 Cisco.IOS. Fix compare version regex for 16.6.5 format.

DLink.DxS

MRTitle
MR6199Fix DLink.DxS.get_mac_address_table script
MR6477DLink.DxS.get_inventory. Set stack number.

Eltex.MES

MRTitle
MR6364add new platform MES3348 Eltex.MES
MR6475Eltex.MES.get_inventory. Add number to stack device.

Eltex.MES24xx

MRTitle
MR6226Fix Eltex.MES24x.get_interfaces script
MR6484Eltex.MES24xx.get_inventory. Add tengigabitethernet for transceiver command.

Generic

MRTitle
MR6401Generic.get_metrics. Add units (Measurement Units) param to set_metrics.
MR6406Generic.get_metrics. Check SNMP for S access preference.
MR6410Generic.get_inventory. Add get_chassis_sensors method for return chassis sensor.
MR6583Generic.get_sla_probes based on DISMAN-PING-MIB example.
MR6587Generic.get_interfaces. Add method check a subinterface

Hikvision.DSKV8

MRTitle
MR6631Hikvision.DSKV8. Add http digest auth.

Huawei.MA5600T

MRTitle
MR6334Huawei.MA5600T.get_metrics. Fix labels format.
MR6570Huawei.MA5600T.get_inventory. Fix typo for detect sensor OID.
MR6624Add support H561SHEA board in Huawei.MA5600T.get_interfaces script
MR6696fix type ethernet interfaces Huawei MA58xx

Huawei.VRP

MRTitle
MR6245Huawei.VRP.get_version - delete \r in version
MR6476Huawei.VRP.get_portchannel. Add cleanup interface name from output.

Juniper.JUNOS

MRTitle
MR6581noc/noc#1898 Juniper.JUNOS.get_interfaces. Fix untagged_vlan typo.

MikroTik.RouterOS

MRTitle
MR6717Fix MikroTik.RouterOS.get_lldp_neighbors.script

NAG.SNR

MRTitle
MR6634Fix prompt match in NAG.SNR profile

NSN.TIMOS

MRTitle
MR6254NCN.TIMOS.get_metrics. Fix DHCP Metrics Used.
MR6386Fix NSN.TIMOS.get_lldp_neighbors script

Qtech.QSW2800

MRTitle
MR6464Fix Qtech.QSW2800.get_version script
MR6518add model QSW2800.get_version

Raisecom.ROS

MRTitle
MR6445fix Raisecom.ROS profile
MR6665Raisecom.ROS.get_version. Add parse_kv_version method for parse version output.

Zyxel.MSAN

MRTitle
MR6601Zyxel.MSAN.get_interfaces. Fix slot count in range.

rare

MRTitle
MR6246Meinberg.LANTIME. Cleanup black.
MR6344Add initial support for 3Com Superstack 3 5500
MR6351fix "configure terminal" promt Eltex.MA4000
MR6414Qtech.BFC_PBIC_S.get_inventory. Add sensor labels.
MR6416Ttronics.KUB. Add sensor labels.
MR6493Angtel.Topaz. Add SNMP support.
MR6497model fix in get_version Huawei.MA5300
MR6532Add initial support for Mellanox.Cumulus profile
MR6568DLink.DAS.get_version. Fix variable 'r' referenced before assignment.
MR6574Linksys.SWR.login. Not used message for NOCError.
MR6580Juniper.EX2500. Fix flake.
MR6600ZTE.ZXDSL98xx.get_inventory. Fix fill_ports method use from profile.
MR6604NSCComm.LPOS. Add SNMP support to scripts.
MR6711Rotek.RTBS. Fix metric units.

Collections Changes

MRTitle
MR6361Add DOM Errors metric.
MR6384Update NAG.SNR profile
MR6435add profilecheckrule SecurityCode.Kontinent
MR6478Remove disposition from 'Security | Authentication | SNMP Authentication Failure' rule.
MR6489Add Eltex.SMG profilecheckrule
MR6517add qtech profilecheckrule
MR6542noc/noc#534 Add DiagnosticCheck to Discovery.
MR6553Fix whitespace in path
MR6591Add Zyxel.MSAN profilecheckrule
MR6635New metrics
MR6636Add compose expression to MetricAction.
MR6649noc/noc#1915 Migrate ManagedObject Attributes to Capabilities.
MR6661Speedup create probe on metrics service.
MR6684Cleanup CDAG subgraph implementation
MR6693add model huawei MA5800-X2
MR6706Add units to 'CPU | Usage' metric
MR6710Replace vcds alarm datasource to vlands.
MR6715update models QSW-8200-28F-AC-DC, QSW-8200-28F-DC-DC

Deploy Changes

MRTitle
MR6241Ansible: fix deprecation warning
MR6291Migrate to Gufo Ping
MR6298Ansible fix pg init command
MR6301change repo clickhouse
MR6342Ansible fix version string type
MR6359Ansible fix apt key proxy problem
MR6368Remove ch_datasource service from deploy
MR6372add cheking certificates for nats and liftbridge
MR6403Fix deploy consul svc-set
MR6436fix tower_run_checks
MR6492Add supporting ubuntu22
MR6537Fix kafkasender checks condition
MR6540change repo for libssl1.1
MR6675fix special character in postgres password
MR6678Rewrite noc_python part of noc role
MR6691Ansible drop debian9 deploy support
MR6700change plugin SimpleJson on simpod-json-datasource
MR6702Add backup slots for metrics
MR6738Fix indent in patch