Version 5.3: JSON Migration and Performance Improvements

• Add new JSON migration mechanism
• Make Service.name a SmallString, otherwise mysql fails to initialize with the following error:
  "MySQLdb.OperationalError: (1170, BLOB/TEXT column 'name' used in key specification without a key length)"
• Increase size of 'payload' field in Rest Call Service
• Performance Optimizations
• Remove lazy join for workflow edges and servers
• Run Targets Devices and Pools association and Run Allowed Targets Update:
• Increase maximum number of threads to 1000
• Move the 'runtime' property to the first column of the Run ('Results') table so that runs are always
  sorted by runtime, even when they have custom names via parameterized form.
• Change memory_size from an Integer to a String in database (to avoid integer overflow issue)
• Add new "scan_folder_on_startup" option to make calling "scan_folder" on first app init optional
• Update bulk edit mechanism to take the bulk filtering form into account when filtering (previously,
  bulk edit does not work with relationship filtering, e.g find all services that belong to a workflow
  then bulk edit filtered services)
• Make 'get_git_content' and 'scan_folder' optional on app startup with new optional "on_startup" key
  in settings.json
• Changelog Diff Improvements:
  • Make 'Side by Side' the default visualization mode and increase panel width
  • Remove the newline in the content of a changelog before 'Added' and 'Removed' so it can be searched in the changelog
    table, and add the diff in Changelog.history so that updates to an object relationships can be displayed in the git
• Fix edge display (workflow edge in workflow builder and link in network builder) not updated after either name or
  color is modified bug
• Fix bug in workflow builder: if user selects node A (single left click), then drags node B (without selecting it),
  then drag node A again, node B is being dragged instead of node A. to fix it, when dragStart is emitted, we check
  whether the node being dragged is selected or not; if it is not, we select it
• Refactor REST Call Service to store payload as string instead of JSON
  • Convert payload field to StringField instead of DictField
  • Add a new property "Substitution Type":
    • String Substitution: the payload is not a valid dict in itself: it will be substituted first, then converted from a
      string to an actual dictionary with ast.literal_eval before using it as "json" argument
    • Dict Substitution: the payload is a valid dictionary represented as a string: it is first converted to an actual
      dictionary with json.loads, then it is substituted (ensures backward-compatiblity)
      By default, the property is set to String Substitution, but migrated services are using the Dict Substitution option.
  • If the 'Subtitution Type' is set to 'Dict Substitution', validate the payload field by converting it to a JSON object
    with json.loads (replaces the "json_only" option of DictField previously)
• Outgoing Edge Mechanism:
  • Refactor the graph traversal algorithm to allow for edges other than success and failure: the user can define
    an "outgoing_edge" variable in python (e.g in post-processing) to decide which edge to follow next
  • Update the Workflow Builder to support displaying multiple edges between two services
  • Add new regression workflow: "Outgoing Edge Mechanism"
• Add 'Export as CSV' buttons in Runs table (Results page) and Tasks table (Scheduling page)
• Fix bug where service targets are not copied when duplicating a workflow (e.g a workflow in SxS service targets mode)
• When refreshing logs with a redis queue, don't always fetch all logs from redis, only fetch the logs up to start_line
  as provided by the front-end
• Add new "size" property to Files to see filesize on disk in form and table
• Normalize strings in the front-end during form serialization to remove windows and mac line endings so that changelogs
  don't get created for line ending changes
• Add new Rate Limiter Feature
• Improve the JS table search code to prevent sending multiple searches in parallel to the server:
• Remove device connection dict from 'connection_cache' in disconnect function if there are no other connections
  for that given, otherwise the connection count in 'check_connection_numbers' is wrong
• Add new 'polling_interval' setting to configure the polling interval of the file monitoring mechanism with
  watchdog PollingObserver
• Decouple the File Monitoring process with watchdog from the main app:
  • The File Monitoring process must be started independently from the main app, with the FILE_WATCHER
    environment variable set to 1
  • Remove the "monitor_filesystem" variable from settings.json
• Don't trigger UI refresh if the tab does not have focus (for table refresh, network builder refresh and
  workflow builder refresh)
• Add new 'result_dict' variable in the results that contain the 'results' before post processing (useful to get
  device output in services like netmiko commands in case post-processing is failing because the 'result' variable
  then becomes the stacktrace) (reported in support slack channel)
• Update 'db.fetch_all': rename the first argument from 'model' to 'instance_type' so that it can be used to fetch
  all devices based on their 'model' property (no more conflict between 'model' first argument and 'model' coming
  from **kwargs)
• Update netmiko command service to always return the result as a list if 'Results as List' option is checked,
  even if there is a single command
• Make Device.ip_address an indexed column
• Refactor "Stop Run" mechanism:
  • Require "run" RBAC access for a user to be allowed to stop a workflow
  • Add new "You don't have permission to stop this workflow." alert in the UI
• Update the REST Call Service with new "proxies" option
• Add session scope when trying to create worker to avoid race conditions when running many
  concurrent calls in parallel

eNMS v4.2

• Add Network builder mechanism
• Add 3D visualization of network devices
• Extend Devices and Links with subclass / custom properties and a separate tab in the UI, the same way services work.
• Remove deep_services function used for export, use service.children relationship instead.
• Dont subclass SQLAlchemy Column following advice of SQLAlchemy creator.
• Make corrupted edges deletion mechanism a troublehooting snippet instead of a button in the admin panel.
• Move redis configuration in settings.json > "redis" key
• Add new mechanism to limit results in server-side drop-down list with filtering constraints.
• Limit superworkflow selection to workflows that contains the shared Placeholder service.
• Set trigger variable to "Regular Run" or "Parameterized Run" when service is triggered from the UI instead of "UI".
• Add SSH Proxy mechanism (multiple jump server in gateways property, gateway device subtype, priority
  tie-break mechanism)
• Consider runtime limiting user / all toggle mechanism in the restart service window.
• Move doc link to settings.json to allow custom doc links for plugins. Generate doc link in the jinja2 template
  instead of javascript (otherwise, wrong doc link until updated in js)
• Move tables refresh rate to settings.json to allow for custom refresh rates.
• New "Category" property / mechanism for the drop-down list of the site and workflow builder.
• Reinstate service selection with single left click (Ctrl no longer needed)
• Remove pytest, coverage, and travis dependencies.
• Reinstate single left click for node selection in workflow & site builder.
• Remove most union subquery in rbac_filter because a union of query yields a CompoundSelect
  SQLAlchemy object, and this is not compatible with using with_entites (via filtering properties kw).
• Fix export in bulk (the hierarchical display mode was not considered, all services inside a workflow
  were exported even when "hierarchical display" was selected)
• Add notification banner mechanism
• Remove default_access property, replace with "admin_only" boolean. Impact on migration.
• Make "run_service" rest api endpoint default to async True
• Update netmiko and napalm Backup services to load deferred row before updating. Impact on both services.
• Remove pathlib from requirements.txt
• Update workflow algorithm to not add services to priority queue in DxD mode if all are discarded.
• Update Ansible Service to use custom path in cwd argument of subprocess.check_output.
• Change default priority to 10 for services. Update of migration files required.
• Implement Cache Invalidation mechanism so that javascript/css files are reloaded at each release.
  Cache invalidation is activatated by setting "invalidate_cache" to true in settings.json.
  eNMS automatically renames the version folder in static / js and static / css to the version number
  stripped from its punctuation.
• Add new check box "Approved by an Admin user" in the Unix Command service. That box must be ticked by
  an admin user for the service to be allowed to run. A non-admin user cannot save a service if it is
  ticked, meaning that each time a Unix Command service is edited, it must be re-appproved.
• Add new timeout parameters for Scrapli service
• Always show security logs, even when logging is disabled. Add "allow_disable" (default: True) keyword argument
  to log function to prevent logs from being disabled if necessary.
• Add new 'deactivate_rbac_on_read' property in rbac.json, under 'advanced' key. Set to true by default.
  When true, eNMS no longers applies rbac for reading from the database. (=> better performances)
• Make the vendor, operating_system and model properties a custom list for devices, links and services,
  and category for sites and workflows. The drop-down list choices can be configured in properties.json > property_list key.
• Add support for renaming objects from the REST API (with key "new_name")
• Add limit to maximum number of nodes that can be displayed in site builder". Configurable via
  visualization.json > Network Builder > max_allowed_nodes
• Add new option to display site nodes as ellipses instead of images for better performances. Configurable via
  visualization.json > Network Builder > display_nodes_as_images
• Auto-update Vendor and Operating System property value of a new service in the workflow builder
  based on the values of these properties in the parent workflow.
• Add support for custom ordering in plugin tables (configurable by overriding the tableOrdering function in the
  table JS class)
• Add support for using device credentials in the Rest Call Service (impact on migration files:
  "username" / "password" => "custom_username" / "custom_password"). Don't allow using device credentials
  if the run method is set to "Run Once".
• Make webssh command configurable from settings / ssh section
• Add new label size property to configure label size in workflow and network builder
• Add new "Configuration" RBAC mode
• Make "sessions" an admin model (visible only to admin users)
• Update git service to support git clone, shallow clone and custom path to local folder (instead of hardcoded path to
  "network_data" folder)
• Update slack notification service to use newest slack_sdk library (instead of slackclient<2)
• Make scrapli connection arguments configurable from automation.json / scrapli / connection_args

eNMS 4

• Refactor of the run mechanism. When running a service, a single run is created and saved to the
  database.
• Remove "operation" (any / all) property from pool
• Change the way pool objects are computed: via SQL query instead of pure python:
  better performances expected for large pools.
• Add regex support for SQLite
• Add new "Invert" option for table filtering
• Move terminal application for web SSH feature inside the application: the terminal application
  was previously moved outside the application because websockets requires sticky sessions which is
  incompatible with having multiple gunicorn workers. Moving to a deployment where eNMS is started
  multiple times with 1 gunicorn worker via the backend stream configuration, it is now possible for
  the terminal to be inside the main application.
• Fix submenu bug when the menu is minimized (gentelella bug)
• Replace prerequisite edge with priority mechanism
• Allow making non-shared service shared and vice-versa (if the shared service doesn't have more than one workflow).
• Separate progress for main devices & iteration devices in workflow builder
• Fix bug where subworkflow device counters not displayed in results when device iteration is used
  Bug report mail: "No status for services in subworkflow with device iteration"
• HTTP requests logging: all requests are now logged by eNMS and not by werkzeug like before.
  => fine grained controlled for what is logged for each request. The log now contains the username.
• Add duplicate button in service table
• Refactor the geographical and Logical View to behave like the workflow builder:

• List of all pools that contain at least one device or link, stored in user browser local storage
• Remove default pool mechanism. Remove "visualization_default" property in pool model.
  By design, the default pool becomes the first pool in alphabetical order
• Add backward / forward control like the workflow builder

• Add global "factory" and "delete" functions in the workflow builder to create and delete new objects
  from a workflow.
• When refreshing a pool, rbac is now ignored so that the pool "refresh" action result does not depend on the
  user triggering it.
• If a workflow is defined to run on a set of devices, and the user lacks access to one or more devices,
  execute for all accessible devices and fail for the inaccessible devices instead of failing the entire workflow.
• Store and commit web SSH session content in backend instead of relying on send beacon mechanism and
  onbeforeunload callback so that the saving of a session does not depend on user behavior
• Add new "Empty" option in table filters and pool definition to filter based on whether the property
  value is empty or not.
• Add table display with property value constraint when clicking on the charts in the dashboard.
• Add scrapli netconf service
• Move LDAP and TACACS+ server init to environment file instead of custom file. Impact on authentication
  ldap / tacacs functions.
• Add Token-based authentication via REST API. New GET endpoint "/rest/token" to generate a token.
• Separate controller (handling HTTP POST requests) from main application (gluing everything together)
  Impact:

• In plugins,
• the "custom" file that contains pre_init, post_init, and the authentication custom code no longer inherits
  from the controller

• Add new "ip_address" field in settings.json > app section
• Add paging for REST API search endpoint: new integer parameter "start" to request results from "start"
• Add server time at the bottom of the menu (e.g for scheduling tasks / ease of use)
• Add button in service table to export services in bulk (export all displayed services as .tgz)
• Ability to paste device list (comma or space separated) into a multiple instance field (e.g service device and pool targets)
• Re-add current Run counter to 'Service' and 'Workflow' on the dashboard banner + Active tasks
• Ability to download result as json file + new copy result path to clipboard button in result json editor panel
• Ability to download logs as text file
• When importing existing workflows via service import, remove all existing services and edges from the workflow
• Upload service from laptop instead of checking for file on the VM
• Add Parameterized Form mechanism to update run properties and payload.
• Add new "full results" button to results tree
• Fix bug in WB where multiple services stay selected
• Add confirmation prompt in workflow builder before deletion
• Change default postprocessing mode to "Run on success only"
• Add log in case postprocessing is skipped
• Add SSH key support in generic file transfer service
• Always set "look_for_keys" to False in generic file transfer service - no longer an option
• Add validation_section mechanism: set path to section of the result to validate (default: results["result"])
• Add new "connection_name" mechanism to open multiple parallel connections to the same device in the
  same workflow
• Add new "get_credential" global variable in workflow builder. Used to get a password or a passphrase
  for a netmiko validaiton command or rest call service. For obfuscation purposes.
  mail: Obfuscate Credentials passed into Netmiko Command Line
• Fix data extraction service and operation keyword in set_var
• Don't set status of currently running services to "Aborted" when using a flask CLI command
• Add TextFSM support for the netmiko validation service (+ regression workflow)
• Add stop mechanism for services in the Result table
• Add server name parameter in Run table to specify which server a service was run from.
  Server to be configured from env variable SERVER_NAME and SERVER_ADDR.
• Lock editing / run of Workflow to group of owners

eNMS 3.22.3

• Add regression workflow for file transfer
• Fix RBAC service run and task scheduling REST API bug
• Fix payload extraction workflow setitem bug
• Add regression workflow with lots of service for scalability testing
• Add regression workflow for skipped service in workflow targets SxS run mode
• Fix rest call service local() scope bug
• Fix get var / set var "devices" keyword bug
• Add jump on connect parameters for netmiko backup service
• Fix skipped query with device in service by service with workflow targets mode bug

eNMS 3.22.2

• Fix iteration device factory commit bug
• Fix workflow in service by service with workflow targets skipped and skip query service bug
• Add missing rbac endpoints in full + read only access
• Fix device creation empty driver due to Scrapli
• Fix workflow iteration mechanism bug

eNMS 3.22.1

• Add user authentication method in user forms
• Fix settings saving mechanism
• Fix gunicorn multiple workers sqlalchemy post fork session conflict bug
• Dont prevent wrong device GPS coordinates from displaying links in network view
• Fix RBAC bugs
• Add new Scrapli service to send commands / configuration to network device

eNMS 3.22

• Remove database url from settings. Configured via env variable DATABASE_URL
• Remote scheduler
• Remove TACACS+ parameters from settings, use env variable instead: TACACS_ADDR, TACACS_PASSWORD
• Make REST API accept Tacacs and LDAP credentials (in the last version, if you were using TACACS+ or LDAP, you could authenticate
  in the UI but couldn't make calls to the REST API)
• Remove LDAP parameters from settings. The LDAP authentication is in the custom controller, there is a default
  function that works with a standard LDAP installation, but you can customize however you want.
  The LDAP server is now configured with the env variable LDAP_SERVER.
  The settings contain a new section "database" to enable ldap, database or tacacs authentication.
• Add replier option in send mail mechanism
• Rename "app_log" option to "change_log" in log function for services
• Add new entry in workflow RC menu "Workflow Results Table": contains all results for a given runtime,
  allowing for comparison of results same device / different service, same service / different device, etc.
• Refactor logging mechanism. In settings.json, add new logging sections to configure whether the log
  for a given logger should also be logged as changelog or service log by default.
• RBAC
• Fix authentication bug flask_login and add session timeout mechanism
• Make plugins separate from eNMS in their own folder, add bash script to install/update/uninstall them
• Make the CLI interface a plugins
• Remove summary from service state to improve workflow refresh performances
• Add Dark mode and theme mechanism
• Make search endpoint work with result to retrieve device results
• Allow dictionary and json as custom properties. For json properties, use jsoneditor to let the user
  edit them.
• Add placeholder as a global variable in a workflow (e.g to be used in the superworkflow)
• Add mechanism for creating custom configuration property
• Refactor data backup services with custom configuration properties. Implement "Operational Data" as
  an example custom property.
• Add new Git service. Replace "git_push_configurations" swiss army knife service with instance of git service.
• Add database fetch/commit retry mechanism to handle deadlocks & other SQL operational errors
• Add validation condition for validation section.

eNMS 3.21.3

• Add new plugins mechanism
• Fix bug help panel open when clicking a field or label
• Add error message in the logs when a service is run in per device mode but no devices have been selected.
• Add default port of 22 for TCP ping in ping service
• Disable edit panel on double-click for start/end services of a workflow
• Fix invalid request bug when pressing enter after searching the "add services to workflow" panel
• Forbid "Start", "End" and "Placeholder" for service names
• Fix Result in mail notification for run once mode
• Make Netmiko prompt command service a substitution string in the UI
• Fix wrong jump password when using a Vault
• Fix workflow results recursive display no path in results bug
• Improve "Get Result" REST endpoint: returns 404 error if no run found, run status if a run is found but there are
  no results (e.g job still running), and the results if the job is done.
• Remove wtforms email validator in example service following wtforms 2.3 release

eNMS 3.21.2

• Fix rest api update endpoint bug
• Add device results to rest api get_result endpoint
• Rename subservice -> placeholder
• Fix rendering of custom boolean properties
• Fix custom properties accordion in service panel
• Fix service cascade deletion bug with service logs and placeholder
• Fix front-end alert deleting services and make it a success alert
• Fix historical config / oper data comparison mechanism
• Fix bug where superworkflow cannot be cleared from list after selection
• Fix bug placeholder service deletion from workflow
• Make superworkflow a workflow property only. Remove superworkflow targets option
• Display only workflows in the superworkflow drop-down list
• Save alert when displaying python error as an alert
• When using a custom logger, only the actual user content is logged
• Update docs rest API
• Improve log function (custom logger behavior / creator)
• Fix superworkflow bug for standalone services
• Dont display private properties in parameterized run results
• Add Ansible playbook service log to security logger
• Update superworkflow initial payload with placeholder service initial payload
• Dont update netmiko and napalm configuration / oper data backup if empty result / no commands

eNMS 3.21.1

• Upgrade JS Panel to v4.10
• Fix jspanel position on long pages with a scrollbar
• Fix subservice double-click bug
• Fix table display bug
• Fix operational data display bug