NOC 23.1


NOC 23.1

23.1 release contains 274 bugfixes, optimisations and improvements.


Topo service

With the 23.1 release, NOC got a new dedicated service for topology-related calculations. The topo service tracks all topology-related changes and maintains an internal graph.

Before the 23.1 release, NOC relied on proper segmentation to calculate uplinks. The uplinks are necessary for topology-based root-cause analysis. We have found that a segment-based approach is hard to implement on specific kinds of networks:

  • Flat networks without the segmentation.
  • Networks with implicit segmentation.
  • Segmented networks without explicit segment hierarchy.

Moreover, it was impossible to build uplinks for top-level root segments.

The new approach analyses the whole network and relies only on managed object levels. The levels are organic and reflect the object’s role in the network. The service tracks changes and analyses all possible paths to exit points. In-memory graph reduces the imposed database load during massive topology changes.

!!! trivia

* `topo` stands for `topology`.
* `un topo` means `a mouse` in italian.

Migrate FM Events to Click House.

Before the 23.1 release, NOC stored the FM events in MongoDB. The limitation of storage became the bottleneck to the system’s scalability.

The lack of collection partitioning in MongoDB didn’t allow us to clean the obsolete data without impact on system operations. The speed of deletion may be lower than the speed of insertion, rendering the implicit deletion or TTL indexes useless. The collection size grew fast. The only working solution was to drop the collection to reclaim the space.

We are working hard on the system performance tuning. The limited MongoDB’s write performance became a stopper.

With the 23.1 release, we have moved the event storage to the ClickHouse and obtained the following benefits:

  • The table partitioning allows maintaining of predictable storage usage by dropping obsolete partitions.
  • ClickHouse has good write scalability.
  • ClickHouse greatly overperforms MongoDB on write operations ever on single-server configurations.
  • It is possible to analyze the events using built-in NOC BI.
  • It is possible to use third-party tools like Tableau for data digging.

Managed Object Workflows

Managed Objects got full workflow integration like other resources. Now the workflow states define the discovery, monitoring, and management settings. The new approach allows greater flexibility and fits well with complex business scenarios.

Configurable Metric Collection Intervals

NOC 23.1 allows configuring different collection intervals for metrics. We also have implemented the collection sharding, which allows multiplexing high-cardinality metrics over time. Metrics collection from boxes with a huge amount of subinterfaces, like PON OLT or BRAS, now is possible. It’s also possible to split metrics depending on the cost of collection on equipment. The “cheap” metrics may be collected frequently, while we can still collect “expensive” metrics more rarely.

Internal Kafka-compatible Message Streaming

NOC now supports Kafka-compatible API for internal message streaming. It’s possible to choose between:

  • Liftbridge, for simple installations.
  • Redpanda for high-profile Linux installations.
  • Kafka for other systems.

NOC supports the deployment and tuning of Redpanda out of the box, and we’re planning to deprecate Liftbridge usage in the next releases.

We also have moved our own Liftbridge client implementation into the standalone Gufo Liftbridge package.

Customized Network Maps

We have reworked the network maps, and now it is possible to create customized maps with the arbitrary set of managed objects. We also have implemented “map generators” on the backend, allowing the auto-generation of custom maps.

New TT Adapter API

We have reworked our TT adapter API. Among the benefits are:

  • Full typing support.
  • Parts of the escalation scenario have been moved into the base classes of the adapter, allowing implementation of the customized scenarios.


FM users must run data conversion scripts manually:

./noc fix apply convert_fm_events
./noc fix apply convert_fm_outages

New features

MR Title
!6805 noc/noc#1942 Customize map backend by loader.
!6883 Add ImageStore for Network Map background files
!6908 noc/noc#1968 Add User Configured Map.
!6942 Move FM events to clickhouse
!6981 noc/noc#1970 Add min_group_size settings for AlarmGroup.
!6990 Add MessageStreamClient for stream work.
!7012 noc/noc#1906 Add RedPandaClient to msgstream.
!7016 noc/noc#2023 New list managed objects
!7024 noc/noc#2022 Add ReportEngine.
!7031 noc/noc#2024 Add interval to Metric Settings.
!7035 noc/noc#2021 Add CPE initial collection and discovery.
!7051 New TTSystem adapter API
!7053 noc/noc#2022 Add Report model.
!7082 noc/noc#2022 Add ReportForm.
!7108 #1865 Add Workflow to ManagedObject.
!7125 topo service


MR Title
!6676 Add script labels
!6728 noc/noc#1928 Correlator add downlink objects for detect ring RCA.
!6749 Add ctl/memtrace endpoint for tracemalloc run.
!6780 Set close escalation delay to reopens alarm control time.
!6781 Update version to 22.2
!6792 Docker add worker, metrics. nginx web volume
!6798 Bump Django version to 3.2.16
!6803 Refactor lib/database_storage module
!6804 Check metrics service active when collected metrics.
!6813 Add bulk mode to set interfacestatus state.
!6820 Use bi_id field as sharding key for Metric Stream.
!6830 Reset ManagedObject diagnostic when disabled Box.
!6831 Check can_update_alarms settings when raise diagnostic alarm.
!6855 Catch ModuleNotFoundError exception when import Windows pyximport library.
!6857 Add apply alarm_class components to raise alarm on correlator.
!6858 Update language translation file.
!6868 Set SNMPTRAP/SYSLOG diagnostics set.
!6888 Fix flake8 ‘l’ error in web service
!6902 Add How-To use hk for collect custom attributes.
!6903 Add NOC shell used examples to doc.
!6904 Add endpoint bulk_ping to activator service
!6911 Add lib to .gitignore and delete lib/
!6913 Update links in welcome screen
!6918 Set SNMP check status on Profile Check.
!6923 Add diagnostic labels.
!6934 Add ObjectDiagnostic Docs.
!6939 noc/noc#1593 Add MapFiled for store BI Events vars.
!6945 Fix ResourceGroup check on alarmescalation.
!6946 Use polars library for Datasource.
!6953 noc/noc#1939 Add service based dcs check params.
!6957 Add sync_diagnostic_labels settings to global config.
!6969 Improve SNMPError description.
!6977 Add ERR_CLI_PASSWORD_TIMEOUT to Authentication Failed.
!6979 Move Stream Config to separate msgstream module.
!6986 Add custom TopologyGenerator settings to UI.
!6996 Increase Map offset for isolated nodes.
!6997 noc/noc#2005 Add selected custom map lookup
!7007 Add InterfaceValidationPolicy check to ConfDB on_delete.
!7007 Add InterfaceValidationPolicy check to ConfDB on_delete.
!7025 Fix EventClass Rules test form
!7027 Fix on_super_password in cli
!7034 Add noc.js to change-ip script path
!7036 Fix network-scan-docs link
!7038 Check pager first on on_prompt script expect.
!7047 Additional AlarmClass to link retention ttl-policy.
!7048 Bump clickhouse version inside docker-compose
!7049 Add DiscoveryIDCachePoison datasource.
!7054 Add site-url for sitemap generation
!7055 Add fm-reboots datasource
!7058 Move change handler to ChangeTracker.
!7063 Add inv-linkdetail datasource
!7064 Update codeowners
!7065 Combine python linters to a single CI task
!7069 Add interval migration.
!7070 Add NoSAProfileError error.
!7071 Catch ResolutionError to RPCNoService.
!7077 Update HP.Comware profile
!7084 Add ttsystemstatds datasource
!7088 Update help command to show custom commands
!7089 Fix create threshold alarms on SLAProbe.
!7093 Bump FastAPI version.
!7094 noc/noc#2045 Bump mongoengine to 0.27 and pymongo to 4.3.3.
!7099 Add rules to MetricConfig on Metrics Service for improve performance.
!7104 Add meta section to metric stream message.
!7105 translation fix
!7106 Make ruff checks visible in joblogs
!7111 Bump pyproj to 3.4.1.
!7112 noc/noc#2046 Bump cachetools to 5.3.0
!7113 noc/noc#2049 Add upload MIB docs.
!7119 noc/noc#2050 Add L2Domain to RemoteSystem model.
!7120 noc/noc#1728 Check labels in match rule when rename and remove
!7123 noc/noc#2022 Migrate Datasource-based tabled report.
!7138 Speedup interface classification.
!7144 #817 Add LAGs interface labels.
!7146 #1539 Set pool_active param default to 1.
!7150 noc/noc#2061 Add error when status is 500 to ManagedObject list
!7152 noc/noc#2060 Add protected field to ManagedObject form
!7153 noc/noc#2063 Add labels to WF Editior State inspector
!7154 noc/noc#2062 Add state combo in filter
!7157 Set Generic.Host as default SA Profile.
!7158 Catch Kafkasender Service connect producer errors.
!7159 Send reboot to BI directly
!7161 Add migrations for allowed_models to Workflow.
!7163 #816 Add inheritance interface profile to aggregate members.
!7164 Remove b" from crashinfo list
!7166 Set icontains to UI State filter condition.
!7170 skip http-exception if status <400
!7172 Add ManagedObject topology DataStream.
!7187 Refactor Diagnostic API.
!7195 Move calculate uplink to TopoService.
!7198 Add labels to setstatus request.
!7200 Cached MetricDiscovery interval.


MR Title
!6747 Fix time_delta when processed discovery metrics.
!6748 Disable suggests in local profile on migration.
!6752 Fix typo on Address.get_collision query.
!6759 Watch escalation when reopen alarm.
!6760 Fix typo on caps discovery logging.
!6763 noc/noc#1936 Fix l2_domain filter on VLAN UI.
!6765 Add send_message method to stub service.
!6770 noc/noc#1937 Fix sender destination send params.
!6775 Fix changelog reorder when compact.
!6777 Split SNMP/CLI credential action on diagnostic discovery.
!6778 Fix check alarm close error on deescalation process.
!6787 noc/noc#1940 Revert Prefix import to Address.
!6789 Fix reorder metrics states on compact procedures.
!6793 noc/noc#1943 Remove vcfilter from NetworkSegment Application.
!6795 Fix partition num on ServiceStub.
!6815 Fix kafkasender stream settings.
!6818 Fix Threshold Profile migration for unique name.
!6822 noc/noc#1785 removed item_frequencies method in fm.reporteventsummary
!6823 noc/noc#1954 Fix wait datastream ready on mx services.
!6827 noc/noc#1955 Add port param to CLI protocol checker.
!6834 Fix allocation order on vlan.
!6845 fix Eltex.LTP get_version
!6849 Fix etl changed labels when object labels is None.
!6854 noc/noc#1956 Fix ZeroDivisionError when prefix usage calc.
!6861 noc/noc#1956 Fix detect address usage with included special addresses.
!6866 Fix send mx message on classifier and uptime reboot.
!6869 noc/noc#1959 Add bulk param to model_set_state.
!6870 Fix typo on NBI objectmetrics.
!6873 noc/noc#1960 Fix error on service without router.
!6882 Fix migration to OS.Linux profile.
!6892 Fix rebuild route chains when delete MessageRoute.
!6900 Fix calculate down_objects metric on Ping Service.
!6909 Fix “no stream jobs” upon collection sync
!6912 Fix OS.Linux profile migration if profile exists.
!6922 noc/noc#1969 Add datastream param to detect changes.
!6926 Add is_delta to _conversions key, for save unit conversation.
!6928 Fix ‘referenced before assignment’ on escalation notify.
!6931 Catch error when transmute processing on Route.
!6943 Fix save in ManagedObject set_caps method.
!6949 noc/noc#1985 Cleanup change commit typo.
!6951 Fix iter datastream typo.
!6954 Fix datastream send message when deleted.
!6962 Fix migrate bi table if previous exists.
!6972 Fix error when change mongoengine DictField.
!6980 noc/noc#1984 Add counter flag to cdag probe for check shift counter type.
!6988 Fix OS.Linux migration for ProfileCheckRule model.
!6989 Fix typo.
!6998 Fix getting slot name on stream config.
!6999 noc/noc#2006 Fix migration threshold profile without function.
!7001 #1998 Bump gufo-ping 0.2.4
!7006 Fix typo portal id on segment map generator.
!7009 fix(peer): issue #2007, as-set format validation and position
!7013 Fix MAC discovery policy filter settings typo.
!7050 Cleanup bad documents on Object Status collection.
!7056 Convert Event Vars to string.
!7080 noc/noc#2039 Fix stucked UI when close tab
!7087 Fix iter_row method on DataSource.
!7090 Fix collection sync for EmbeddedDocumentListField.
!7092 noc/noc#2041 Sync cursor after flush state on MetricServce.
!7098 Fix aoikafka requirements.
!7102 noc/noc#2047 fix me.up() is undefined
!7102 noc/noc#2047 fix me.up() is undefined
!7114 Fix typo on MessageRoute UI Form.
!7121 Fix wipe user command.
!7122 Fix Events log.
!7124 noc/noc#2054 Fix rebuild datastream on DNS Model.
!7129 Fix DNSZone datastream when IP address used on masters.
!7142 Fix classifier Event Message format for send to
!7148 noc/noc#2059 Catch getting error for MAC Collection button
!7149 Slice activator script result publish for large result size.
!7151 Fix msgstream client for migrations.
!7168 Rebuild managedobject datastream when changed discovery id.
!7173 #2065 Place interface IP Addresses to object VRP if device not supported VRF.
!7183 Use Generic.Host profile for unknown peering point SA profile.
!7189 Fix liftbridge client alter stream.
!7194 Fix getting external stream partition on Router.
!7196 Fix error when getting datastream format message headers.
!7197 Fix csvutil processed import.
!7199 noc/noc#2068 Disable clean when collection sync for instances without uuid.

Code Cleanup

MR Title
!6800 Refactor lib/highlight module
!6801 Refactor lib/template module
!6802 Remove lib/datasource module
!6829 Move lib/app directory into services/web/base
!6987 Cleanup print on config class.
!7052 Ruff linter
!7062 Simplify mib expressions
!7072 devcontainer.json: Move settings and extensions into customizations.vscode
!7073 ruff: Enable W - pycodestyle warnings
!7074 ruff: Enable flake8-builtin (A) diagnostics
!7075 Ruff: Enable pylint (PLC, PLE) checks
!7078 ruff: Fix PLW0120 else clause on loop without a break statement
!7134 Catch git error when getting version.

Profile Changes


MR Title
!6810 Alstec.24xx.get_metrics. Fix metric units.


MR Title
!7117 noc/noc#1920 Cisco.IOS. Cleanup output SNMP CDP neighbors.


MR Title
!7059 Cisco.IOSXR get_inventory error asr9k
MR Title
!7103 DLink.DxS.get_interfaces: Fix CLI returns wrong oper_status


MR Title
!7147 Add Dahua.DH profile to collection
!7147 Add Dahua.DH profile to collection


MR Title
!6915 Eltex.MES. Add retry authentication to pattern_more.
!6965 fix interface description Eltex.MES.get_interfaces
!6974 Eltex.MES. Add MES-3316F and MES-3348F oid.
!7004 fix Stack Members in get_capabilities Eltex.MES
!7026 Eltex.MES. Add MES-2348P to detect oid version.
!7041 fix get_inventory Eltex.MES. Serial fix
!7041 fix get_inventory Eltex.MES. Serial fix
!7066 inv.platforms: Eltex MES-2324FB
!7068 mes2324fb
!7097 fix portchannel Eltex.MES


MR Title
!6842 Fix Eltex.MES24xx.get_version script


MR Title
!6746 Use Attribute capability for get_inventory scripts.
!6896 Generic.get_capabilities. Filter non-printable character on sysDescr.
!6959 Generic.get_interface_status_ex. Ignore unknown interface on interfaces param.
!6959 Generic.get_interface_status_ex. Ignore unknown interface on interfaces param.
!6964 add chunk_size to Generic.get_interfaces
!7155 noc/noc#1983 Add return script execution metrics on Activator.script.
!7165 Fix units on collecting SLA metrics on profiles.


MR Title
!7137 Hikvision.DSKV8. Fix NTP Server parse on ConfDB normalizer.
!7137 Hikvision.DSKV8. Fix NTP Server parse on ConfDB normalizer.


MR Title
!6783 noc/noc#1926 Huawei.MA5600T. Fix allow_empty_response for pattern_more send.
!7037 noc/noc#2020 Huawei.MA5600T.get_inventory. Fix detect board.
!7100 Fix CPE discovery
!7192 #2056 Huawei.MA5600T.get_inventory. Fix duplicate chassis as motherboard on MA5801-GP16.


MR Title
!6799 Fixed detect port and power supply number for new Huawei CloudEngine switches
!6895 noc/noc#1964 Huawei.VRP.get_interfaces. Add allow_empty_response for ‘display vlan’ on cloud_engine_switch.


MR Title
!6833 Juniper.JUNOS.get_metrics. Fix units on ‘Memory | Heap’ metrics
!6850 Juniper.JUNOS.get_metrics. Fix labels format on slot generator.
!7101 Juniper.JUNOS.get_metrics. Fix collect SLA metrics.


MR Title
!7128 noc/noc#1914 MikroTik.RouterOS. Fix config normalizer when router destination is ifname.
!7128 noc/noc#1914 MikroTik.RouterOS. Fix config normalizer when router destination is ifname.


MR Title
!7060 fixing NAG.SNR.get_inventory


MR Title
!6767 Fix Raisecom.ROS.get_version script


MR Title
!7115 noc/noc#1658 ZTE.ZXA10.get_interfaces. Add SFUL, GFGM card type.


MR Title
!6769 Fix 3Com.SuperStack3_4500.get_interfaces script
!6807 DCN.DCWL.get_metrics. Convert to flot.
!6825 DCN.DCWL.get_metrics. Fix check ‘channel-util’ key in metrics.
!6825 DCN.DCWL.get_metrics. Fix check ‘channel-util’ key in metrics.
!6884 Fix Qtech.QSW.get_version script
!6897 ECI.HiFOCuS. Fix setup_script profile method for None user.
!6961 H3C.VRP.get_interface_status. Fix matchers typo.
!6976 Cambium.ePMP. Add SNMP support.
!6995 Eltex.WOP. Add SNMP support.
!7019 add get_lldp_neighbors Qtech.QOS
!7030 DLink_Industrial_cli Fix (config) prompt and autoanswer
!7086 fix Zyxel.DSLAM
!7130 noc/noc#2037 BDCOM.xPON.get_interfaces. Add Giga-Combo-FX-SFP interface type.
!7132 Fix P1 interfaces on port1 Qtech.QOS
!7132 Fix P1 interfaces on port1 Qtech.QOS
!7143 #2037 BDCOM.xPON.get_interfaces. Fix parse tagged vlans.
!7180 Расхождение коллекции

Collections Changes

MR Title
!6837 inv.platforms: Huawei Technologies Co. S6730-H24X6C
!6838 inv.platforms: Huawei Technologies Co. S6330-H48X6C
!6839 inv.platforms: Huawei Technologies Co. S6330-H24X6C
!6885 Fix calculate MetricType for delta type.
!6914 Fix ComboPorts on ObjectModels.
!6936 ping: Switch to direct dispose protocol
!6993 noc/noc#1958 Add bulk mode for update object statuses on dispose message.
!7040 add profilecheckrules SKS-16E1-IP-ES-L
!7042 noc/noc#1729 Replace AlarmClass default severity by AlarmRule and labels.
!7079 noc/noc#2013 Add buckets to iter_collected_metrics for discovery.
!7085 add profilecheckrules zyxel.dslam VES-1624FT-55A
!7109 #2022 Add report config

Deploy Changes

MR Title
!6541 Add redpanda role deploy
!6736 add lib yedit
!6877 Ansible tower add metrics check
!7061 Split requirements.txt
!7076 Ruff: Enable pylint (PLR) checks