v4.1.0 Release Notes

KubeSphere Enterprise v4.1.0 Release Notes


## Main Features

### Lightweight

- Remove unnecessary extensions.
- Support fast deployment and uninstallation via helm.

### Extensions

- Take a new microkernel architecture - Luban, which support extending functions by various extensions.
- Provide online KubeSphere marketplace for getting extensions, including free ones.

### New UI

- A brand-new console.
- Provide global injection points for extensions.

### Multiple Clusters

- Simplify the deployment of agents in member clusters.
- Support importing member clusters via kubeconfig.


### Project Management

- Support managing workloads such as Deployment, DaemonSet, StatefulSet, Job, and CronJob.
- Support managing resources such as Secret, ConfigMap, and ServiceAccount. 
- Support managing resources such as Service, and Ingress.
- Support managing resources such as PVC.
- Support Quota, LimitRange and other configurations.

### Workspace Management

- Support cross-cluster project management.

### Cluster Management

- Support node management: node eviction, taint management, etc.
- Support cluster project management.
- Support cluster resource management: Deployment, DaemonSet, StatefulSet, Job, CronJob, Secret, ConfigMap, ServiceAccount, etc.
- Support the management of StorageClass and PresistentVolumeClaim.

### DevOps

- Support creating independent DevOps projects and access-controlled CI/CD pipelines.
- Easy to start without complicated Jenkins configuration.
- The Jenkinsfile-based pipeline provides a consistent user experience and supports multiple code repositories.
- Easy to create pipelines on the graphical panel.
- Powerful tool integration, such as SonarQube, for code quality checking.

  
## Limitations
Upgrade from version 3.x to 4.x is not supported at present, but will be supported in subsequent releases.

The following functions are temporarily unavailable and will be offered by extensions later:

- Monitoring
- Alerting
- Notifications
- Istio
- Spring Cloud
- Project gateway and cluster gateway
- Storage volume snapshot
- Network isolation
- Whizard observability center
- RadonDB DMP

The following features are currently unavailable and will be supported in subsequent versions:

- User group management
- OpenPitrix for app management
- Build images through S2I / B2I in DevOps

v4.0.0 Release Notes

KubeSphere Enterprise v4.0.0 Release Notes


## DevOps

### Enhancements & Updates

- Support for customizing the Jenkins service types in the Installer configuration, as well as optimizing the timeouts of readiness and liveness probe for the ArgoCD service.
- Merge DevOps-related alerting rules into a group.
- Upgrade Jenkins to version 2.346.3, and upgrade some plugins.
- Add a plugin to get the build user information in Jenkins and add this step to the graphical pipeline panel.
- Refactor the DevOps interface to avoid getting data directly from jenkins.

### Bug Fixes

- Fix the request branch interface bug of DevOps single branch pipeline.
- Fix the discover branches default value for git type of DevOps multi-branch pipeline.
- Fix the issue that shell commands cannot be saved when editing the pipeline graphically.
- Fix the issue that the pipeline run log is too large to view online.
- Fix the error when replaying a pipeline.
- Fix the layout of the DevOps runlog page.
- Fix the display issue of DevOps projects under Recent Acess.


## Multi-cluster

### Enhancements & Updates

- Validate cluster roles when adding clusters.
### Bug Fixes

- Fix webhook validation issue for new clusters.
- Clean up clusters only when a cluster is ready.
- Fix the issue that cluster authorization is not revoked after deleting a cluster.
- Fix the issue that the added tag is not shown after adding a cluster.

## Observability

### Features

#### Monitoring

- Add the calico-exporter component to provide monitoring metrics for Calico, including BGP, IPPool, and IPAM.

#### Alerting

- Add support for filtering alerts by keywords.
- Support adding labels when adding/updating rules.

#### Logging

- Export logs in positive or reverse order.

#### Notification

- Update notification-manager to support filtering notifications using regular expressions.
- Support multiple receivers, and regular expressions for notification filtering/silencing.

### Enhancements & Updates

#### Monitoring

- Add Whizard server params in the ClusterConfiguration file.
- Optimize process-exporter configuration to improve its responsiveness.
- Update blackbox-exporter to version 0.24.0.
- Change the severity of alerting rules to `info` if it is `none` or empty.
- Update Whizard to version 0.9.0.
- At least one cluster should be selected when selecting clusters in Whizard.
- Remove the "Usage" information from "Cluster Quota Statistics" on the overview page of a cluster.

#### Alerting

- Increase CPU limit for default-http-backend to avoid CPUThrottlingHigh alerts.
- Add a `NodeExporterDown` rule to the built-in `node-exporter` GlobalRuleGroup, and configure kube-state-metrics to export `kube_node_info` metric by default.
- Increase CPU limit for some monitoring components to avoid CPUThrottlingHigh alerts.
- Support configuring alerting rules at the `Info` level. 
- Add calico-related alerting rules.
- Show full names of monitoring targets in alerts.

#### Logging

- Optimize parameters of fluentbit to improve log collection efficiency.
- Support automatic update of usernames and passwords of external Elasticsearch or OpenSearch clusters.
- Add a switch to turn on/off Elasticsearch/OpenSearch curator.
- Add a switch to disable output logs to Elasticsearch/OpenSearch.

#### Notification

- Update notification-manager to v2.4.0.

#### Auditing

- Update kube-auditing to v0.3.0.


### Bug Fixes

#### Monitoring

- Fix the storage statistics issue of workspaces.
- Fix the scope of rules when configured as template rules in GlobalRuleGroups.
- Fix the issue that some target down rules cannot alert as expected in multi-cluster.
- Optimize the PVC monitoring prompt.

#### Alerting

- Fix the issue that when a member cluster is removed, alert messages from that member cluster are still shown in Whizard.
- Fix alerting rules created for a cluster still exist when the cluster is removed.
- Update alerts statistics promQL.
- Fix unexpected duplicate notifications when the notification channel is configured with a cluster filter in multi-cluster. 
- Fix the issue that cannot get the system-workspace of a cluster when creating a custom alerting policy.
- Fix the issue that labels are required when creating an alerting policy.
- Fix the issue that rule names are not shown in alerting messages under clusters and projects.
- Fix the issue that project names are not filled automatically in the monitoring target of alerting rules.

#### Notification

- Fix the missing notification history issue when the container runtime is contained.


## Storage

### Features

- Storage usage can be categorized by storage type.

### Enhancements & Updates

- Remove GlusterFS and Ceph RBD options when creating storage classes.

### Bug Fixes

- Fix the issue that storage class authorization rules are not working in some cases.
- Fix storageclass-accessor webhook rules.

## Gateway

### Bug Fixes

- Fix the checksum error when updating the workspace gateway if no gateway namespace is configured.
- Fix the permission error when viewing the workspace gateway by an account with regular or self-provisioner permission on a workspace.
- Fix the permission error when viewing the monitoring info of the workspace gateway by an account with regular or self-provisioner permissions on a workspace.

## DMP

### Enhancements & Updates
 
- OpenSearch: add Logstash configuration and standalone monitoring and alerts.
- OpenSearch: upgrade from 2.3.0 to 2.8.0.
- Kafka: support user-specific flow and speed limits, and add description in permission rules.
- Kafka: the default certificate validity is adjusted to 10 years.
- Kafka: support increasing the number of partitions in a created topic.
- Kafka: add the Jmx parameter, which supports opening the Jmx port for cmak.
- Upgrade Kafka Operator to version 0.2.0 and OpenSearch Operator to version 2.0.25, remove ECK and ClickHouse apps, and remove Redis Cluster & Standalone 6.2.5.
- OpenSearch, Kafka, and Redis support preset cluster alerts.

### Bug Fixes   

- OpenSearch: fix PromQL for some monitoring metrics.
- OpenSearch: change Logstash to Statefulset to ensure that monitoring information can still be obtained even if pods are recreated.


## User Experiences

### Features

- Support filtering pods by pod IP.
- Support displaying the LDAP login page via URL.
- Pods in the pod list can be filtered by pod IP.

### Enhancements & Updates

- Change the default resync period of Informer to 10 hours.
- When a license expires, the notification history page and the user notification settings page are still available as before.


### Bug Fixes

- Fix global alerting styles of Whizard.
- Fix the issue that the added container image is incorrect if not setting the image tag when creating a workload.
- Fix the issue that container images are unsearchable when editing workloads.
- Show status when editing crd info.
- Fix the failure when deploying from an app template without values.yaml.
- Fix the issue that the admin info is not synchronously cleared after clearing the workspace info in the window for allocating projects to workspaces.


## Authentication & Authorization

### Enhancements & Updates

- Add resource configuration for Redis, OpenLDAP, and metrics-server.

## Network

### Enhancements & Updates

- Update `namespaceNetworkPolicy` crd to support user-defined port ranges. (Only back-end support)

v3.5.0 Release Notes

KubeSphere Enterprise v3.5.0 Release Notes


## DevOps

### Enhancements & Updates

- Add support for reusing the created git repository on the pipeline edit and creation page.
- Optimize the layout of the pipeline run log page.
- Optimize the devops-jenkins JVM configurations.
- Add a tooltip for the replay function.
- Support the email extension in pipelines.

### Bug Fixes

- Fix an issue where the deleted branch pipeline can still be triggered.
- Fix an issue where the page crashes if users enter a non-existent project when creating a CD task.
- Add ArgoCD for continuous deployment to the system project.
- Fix an issue where deployment resources are not automatically deleted when deleting a CD application.
- Fix an issue where downloading artifacts for multi-branch pipelines fails.
## Observability

### Enhancements & Updates

- Remove misleading namespace and pod labels from node alerts.
- Add relevant PVC info to PV alerts.
- Upgrade Whizard to v0.7.0.
- Optimize the PrometheusRule sync logic in multi-cluster scenarios.
- Add the httpClientConfig option to the alerting and monitoring to support HTTPS connection to other components.
- Add support for setting the maximum number of exported logs.
- Add support for querying notification history by status.

### Bug Fixes

- Fix ETCD monitoring configuration issues.
- Fix the Infohibitor rule.
- Fix an issue where node_timex_sync_status does not exist.
- Fix an issue where metrics used by some built-in alert rules do not exist.
## App Store

### Bug Fixes

- Fix an issue where some applications cannot be installed in the App Store.

## Multi-tenancy and Multi-cluster

### Bug Fixes

- Fix an issue where cluster names are not deduplicated.

## Network

### Bug Fixes

- Fix an ippool block validation issue.

## Security

### Enhancements & Updates

- Add support for automatically injecting pod security configurations.

## Authentication and Authorization

### Bug Fixes

- Fix an issue where an IdentityProvider conflict occurs after modifying the configuration file.
- Fix an issue where enableMultiLogin configurations do not take effect.

## DMP
### OpenSearch
#### Enhancements & Updates

- Added JVM adaptive settings. If the value of JVM is not specified, it will be set to half of the container memory.
- Introduce the rescue mode, allowing users to restart the cluster when it's in a non-green state.
- Add support for integrating with third-party identity authentication OIDC.
- Increase the maximum disk specifications to 15 TB.

#### Bug Fixes
- When the storage type does not support scaling, a prompt will be displayed to inform users that the scaling operation cannot be performed.
- Fix an issue where monitoring data was incomplete after the IP address of a pod changes.

### Redis Cluster and Redis Sentinel
#### Enhancements & Updates

- Add the adaptive `maxmemory` parameter. If the value of `maxmemory` is not specified, it will be set to 95% of the container memory.
- Support log level adjustment and optimize log content.
- Improve security by encrypting passwords.

#### Bug Fixes
- Fix an issue where the cluster status keeps switching between **Updating** and **Running** after an upgrade.
- Fix an issue where monitoring data cannot be retrieved after the password changes.

## User Experience

- Fix an issue where the deletion logic of workloads and services are inconsistent.
- Removed deleted cluster tags from workspace lists.
- Fix the project name and repo name rule of Harbor.
- Fix the cron validation error for the custom scaling (hap) in the start time and end time.
- Fix an issue where resource quota cannot be changed.
- Fix an issue where an error occurs while searching for an image after a user sets the image service address to **https://docker.io**.
- Add support for searching for and selecting tags.

### API Changes

- Add the `checkDevopsName` and `checkPipelineName` APIs for validating whether DevOps project and pipeline names are duplicates.
- Add `/kapis/monitoring.kubesphere.io/v1alpha3/validation/query API endpoint` that validates PromQL expressions.
- Deprecate the Configz field in cluster status, which will be removed in the next major release.

v3.4.1 Release Notes

KubeSphere Enterprise v3.4.1 Release Notes


## DevOps

### Features

- Add support for defining pipeline steps by using CRDs.
- Save PipelineRun results to configmap.

### Enhancements & Updates

- Optimize editing of pipeline process and add support for asynchronous invocation of Jenkins' toJson and toJenkinsfile APIs.
- Display CI progress in git provider.
- Filter pipelines and multi-branch pipelines by category.
- Upgrade Subversion-related plugins of Jenkins.

### Bug Fixes

- Fix an issue where Webhook configurations are lost after users modify and save pipeline configurations.
- Fix an issue where the status of PipelineRun changes from "Cancelled" to "Not Running".
- Fix an issue where a failure is triggered each time DevOps pipeline controller fails to retrieve information from Jenkins.
- Fix an issue where new users are unable to run pipelines due to mismatched keys.
- Fix an issue where the value of the retention period and the maximum build records must be an integer on the pipeline editing page.

## Storage

### Features

- Add support for setting storage resource quotas.
- Add support for managing storage quotas.

### Bug Fixes

- Limit expansion by using storage quotas.
- Update Webhooks related to storage quotas.
- Fix an issue where the binding status in the PV list is incorrect.

## Service Mesh

### Features

- Upgrade Istio to v1.14. Kiali to v1.50，and Jaeger to 1.29.

### Bug Fixes

- Fix an issue where enabling Jaeger fails when a jaeger-secret error is reported.

## Observability

### Features

- Add support for configuring Whizard parameters in the KubeSphere Whizard Observability Center Console.
- Support autoscaling based on custom metrics or custom triggers from Prometheus, Kafka, and Cron.
- Integrate processor exporter.
- Integrate blackbox-exporter.

### Enhancements & Updates

- Optimize the display of alert rules on the Whizard Observability Center page.
- Optimize the display of the monitoring page when there is no data.
- Replace built-in Elasticsearch with built-in OpenSearch as the log storage backend.
- Upgrade logsidecar-injector to v1.2.0.
- Upgrade Notification Manager to v2.3.0.
- Upgrade kube-events to v0.6.0.
- Upgrade Whizard to v0.6.1.
- Support Prometheus authentication configurations.
- Upgrade OpenSearch and OpenSearch Dashboard to v2.6.0.

### Bug Fixes

- Fix an issue where the resource usage ranking page is frozen.
- Fix an issue where applicable clusters of built-in alert rules cannot be modified.
- Fix an issue where alert targets are displayed inaccurately.
- Fix an issue where alerts are not triggered when the number of alert rules exceeds the limit.
- Fix the alert module enabling issue.

## KubeEdge Integration

### Enhancements & Updates

- Upgrade KubeEdge to v1.13.0.

## Network

### Enhancements & Updates

- Optimize IP pool parameters.

### Bug Fixes

- Fix the locales file of IP pools.
- Modify the migration rule for migrating pods of a disabled IP pool to another pool.
- Modify the Overlay rule when creating an IP pool.

## Microservices

### Features

- Add affinity, tolerations parameters for gateway.
- Support workspace gateway.
- Add the default value for worker-processes.

### Enhancements & Updates

- Upgrade nginx-ingress to v1.3.1.

### Bug Fixes

- Fix an issue where cluster gateway data is duplicate.
- Fix an issue where cluster information is not displayed in the cluster gateway data.

## App Store

### Enhancements & Updates

- Add independent authentication API for Identity Provider of the Password type.
- Add the IAM v1beta1 API.

### Bug Fixes

- Fix an issue where an application fails to be reinstalled after users initially fail to install the application, and then change configurations of the application.
- Remove authentication APIs marked as Deprecated in KSE v3.0.0.
- Mark IAM v1alpha2 API as Deprecated.

## Authentication & Authorization

### Features

- Add an authorization status prompt on the license page.
- Add support for hiding unauthorized components on the license page.
- Add a page for viewing the end-user license agreement.

### Enhancements & Updates

- Require that the minimum password length must be 8 characters.

## Security

### Enhancements & Updates

- ks-core runs as non-root user and uses a more secure base image.
- Fix a potential heapdump file leakage vulnerability in kubesphere/spring-cloud-gateway.
- Users are forced to change initial passwords upon first login.
- Provide more complete security materials: SBOM based on SPDX standards and open source software list.

## DMP

### OpenSearch

#### Enhancements & Updates

- Upgrade operator to v2.0.19 and chart to v2.0.21.
- Support NodePort for Service.
- Support configurations of log output formats.

#### Bug Fixes

- Fix verification and conversion issues of the disk unit.
- Add the kubesphere tag to properly display data in specific monitoring scenarios.
- Fix the issue that monitoring data is not displayed in multi-cluster scenarios.
- Fix the log timestamp issue.

### MongoDB

#### Enhancements & Updates

- Upgrade chart to v2.0.8.

#### Bug Fixes

- Fix verification and conversion issues of the disk unit.
- Add the kubesphere tag to properly display data in specific monitoring scenarios.

### Redis

#### Enhancements & Updates

- Upgrade chart to v2.0.8.
- Enhance high availability of Sentinel.

#### Bug Fixes

- Fix the issue that the namespaceOverride parameter does not take effect.
- Change the image pull policy from Always to IfNotPresent.
- Add the kubesphere tag to properly display data in specific monitoring scenarios.

### MySQL

#### Bug Fixes

- Fix the issue that the imageNamespaceOverride parameter in the Helm Chart does not take effect, resulting in image replacement failures.
- Fix the incorrect permission in the configuration file in MySQL 8.0.

### PostgreSQL

#### Enhancements & Updates

- Upgrade chart to v1.2.0.
- Add features, such as creating backups and scheduled backups, plus recovery.
- Update the CRD field.

#### Bug Fixes

- Add the kubesphere tag to properly display data in specific monitoring scenarios.

### Kafka

#### Enhancements & Updates

- Upgrade chart to v 0.1.1 and remove unused YAML for metric statistics.
- Add the num.partition parameter.
- Optimize the topic naming rule.

#### Bug Fixes

- Fix an issue that the network policy is enabled by default.
- Fix an error stating that the secret cannot be obtained after Kafka is deployed.

## User Experience

- Support renaming of clusters, projects, workspaces, and DevOps projects.
- Optimize some UI texts and description.
- Fix an issue where alias search is not supported.
- Fix an issue where GPU is not limited when users are creating a service and application form on the container settings page.
- Fix an issue where URL error is reported after the system redirects from the grayscale release details page of projects to the container details page.
- Fix an issue where the length of cluster names is too long.
- Optimize the visibility list and popup style for clusters.
- Fix an issue where the cluster to which system-workspace belongs is not displayed.
- Fix an issue where an error is reported on the cluster details page after the cluster name is changed.
- Modify the alias matching regex.
- Fix the form style for Webhook URLs.
- Support pod grace periods.
- Optimize the LDAP login page.
- Support weighted aliases.

## API Changes

### Deprecated APIs

When APIs evolve, some old APIs are deprecated and eventually removed. The following APIs have been marked as Deprecated and will be removed in the next major release.

- All of alpha resource API (**iam.kubesphere.io/v1alpha2**).
- The alpha `Authentication` API(**/kapis/iam.kubesphere.io/v1alpha2/login**).
- All of alpha resource API (resources.kubesphere.io/v1alpha2).
- All of alpha resource API (resources.kubesphere.io/v1alpha3).

### Removed APIs

The v3.4 release will stop serving the APIs we've listed below. These are all beta APIs that were previously deprecated in favor of newer and more stable APIs.

- The alpha `tojenkinsfile` API (**/kapis/devops.kubesphere.io/v1alpha2/tojenkinsfile**).
- The alpha `tojson` API (**/kapis/devops.kubesphere.io/v1alpha2/tojson**).
- The alpha `Authentication` API (**/kapis/iam.kubesphere.io/v1alpha2/login**).


v3.4.0 Release Notes

KubeSphere Enterprise v3.4.0 Release Notes


Starting from v3.3.1, the previous QingCloud KubeSphere Container Platform (QKCP) has been renamed to KubeSphere Enterprise.

## Observability

### Features

- Add Whizard v0.5.0 as the Prometheus long-term storage for multi-cluster monitoring & alerting.
- Add the Observability Center, which supports multi-cluster monitoring and alerting.
- Introduce three new CRDs `GlobalRuleGroups`, `ClusterRuleGroups`, and `RuleGroups` for multi-cluster, cluster, and project alerting respectively. Alert rules are now defined in `GlobalRuleGroups`, `ClusterRuleGroups` or `RuleGroups`, which will be converted to groups in `PrometheusRules`.
- Add a controller to sync `PrometheusRules` to the host cluster.
- Add an admission webhook and controllers for `RuleGroups`, `ClusterRuleGroups`, and `GlobalRuleGroups`.
- Built-in alert rules can be modified, enabled, disabled, and reset now.
- Add support for alerting of KubeSphere core components.
- Add support for cleaning up a cluster's notification settings when removing this cluster.
- Add the Feishu notification channel.
- Add support for silencing notifications.
- Add support for changing the language of notifications.
- Add support for adding a user account's email address to this user's notification settings.
- Use OpenSearch as built-in log storage instead of Elasticsearch.
- Add the OpenSearch curator.
- Add the OpenSearch Dashboard.
- Add OpenSearch as the log receiver.
- Add support for searching logs from OpenSearch in the KubeSphere console.
- Add process and thread monitoring metrics for containers.
- Add disk monitoring metrics, for example, disk usage data.
- Add support for importing Grafana templates to create a custom monitoring dashboard in a namespace.
- Add support for defining data retention periods for container logs, Kubernetes event logs, and audit logs respectively.

### Enhancements & Updates

- Improve the cluster overview page.
- Improve workspace and project monitoring.
- Upgrade Prometheus to v2.39.1, Thanos to v0.29.5 (customized), and kube-state-metrics to v2.6.0.
- Upgrade fluentbit-operator to v0.14.0.
- Upgrade Notification Manager to v2.1.0.
- Upgrade Alertmanager from v0.21.0 to v0.23.0.
- Upgrade Grafana from v7.4.3 to v8.3.3.
- Upgrade node-exporter from v0.18.1 to v1.3.1.
- Upgrade Prometheus Operator from v0.43.2 to v0.55.1.
- Upgrade kube-rbac-proxy from v0.8.0 to v0.11.0.
- Upgrade configmap-reload from v0.3.0 to v0.5.0.
- Upgrade kube-events from v0.3.0 to v0.4.0.
- Upgrade Fluent Bit from v1.8.3 to v1.9.4.

### Bug Fixes

- Fix the style of log details in the toolbox.
- Fix inaccurate time unit of the monitoring metrics.
- Fix the "No Cluster Available" issue during log search.

## DevOps

### Features

- Add the Continuous Deployment feature, which supports GitOps and uses Argo CD as the backend.
- Optimize pipeline templates loaded from APIs.
- Add the allowlist feature on the **Basic Information** page of a DevOps project.
- Add the **Events** tab on the pipeline run record page.

### Enhancements & Updates

- Add support for editing the kubeconfig binding mode on the pipeline UI.
- Add a message indicating that the S2I and B2I features currently do not support the containerd runtime.

### Bug Fixes

- Fix an issue where Jenkins updates are not synchronized in real time.
- Fix the cron expression check failure of Jenkins.
- Fix an issue in the **Import Code Repository** dialog box, where repositories and organizations are not completely displayed.
- Fix an issue where users fail to check the CI/CD template.
- Remove the `Deprecated` tag from the CI/CD template and replace `kubernetesDeploy` with `kubeconfig binding` at the deployment phase.
- Fix an issue where pipeline parameters are not updated in time.
- Fix the automatic cleaning behavior of NoSCMPipeline, and keep it consistent with the cleaning configuration of Jenkins.

## Multi-Tenancy & Multi-Cluster

### Features

- Add support for grouping cluster by labels.
- Add a limit on the number of labels that can be added to a cluster.
- Add support for automatically renewing a cluster's kubeconfig.
- Add metrics for cluster certificate expiration. Users will be notified when the kubeconfig is about to expire in 7 days or 24 hours.
- Add support for removing the cluster warning message.
- Add the name of current cluster to the `kubesphere-config` configmap.
- Support cluster member management and cluster role management.

### Bug Fixes

- Set the ready status to false when a cluster fails to join the federation control plane.

## Microservice

### Features

- Support Spring Cloud and add Spring Cloud Operator.
- Support Spring Cloud configurations and Spring Cloud Microservice in Spring Cloud Operator.
- Support Spring Cloud Gateway in Spring Cloud Operator.
- Support Microservice Aggregated API in Spring Cloud Operator.
- Add GatewayRouteConfig in Spring Cloud Operator.
- Add the springcloud-controller helm chart.
- Add kubesphere-spring-cloud-starter.

## Storage

### Features

- Support PVC resizing, which can automatically expand PVCs when the remaining capacity is insufficient.
- Support volume snapshot content management and volume snapshot class management.
- Allow users to set authorization rules for storage classes so that storage classes can be used only in specific projects.
- Provide usage data of each disk.

### Bug Fixes

- Set `hostpath` as a required option when users are mounting volumes.
- Update storageClass-accessor, so that deleting storage resources no longer requires accessor validation.

## Authentication & Authorization

### Features

- Add support for manually disabling and enabling users.

### Enhancements & Updates

- Check user permissions to clusters when users are creating or updating workspace templates.

### Bug Fixes

- Add support for filtering workspace roles using the LabelSelector.
- Add support for customizing or randomly setting an initial admin password.
- Delete annotations in `role-template-manage-users`, `role-template-view-members` and `role-template-manage-roles`.
- Fix an issue where `cluster-admin` cannot view and manage the configmap, secret, and service account.
- Delete role `workspace-manager`.
- Add role `platform-self-provisioner`.
- Delete role `users-manager`.
- Block `role-template-manage-groups`.

## DMP

### Features

#### DMP Framework

- Refactor monitoring by adding application monitoring information.

#### ClickHouse

- Support ClickHouse kernel version 21.1.3.32.
- Support more ClickHouse metrics.
- Support horizontal and vertical expansion.
- Support SVC-based cluster load balancing.
- Support ClickHouse HA.
- Support more cluster states.
- Support disk resizing with the QingCloud CSI.
- Add support for automatically provisioning and managing ZooKeeper clusters.

#### MySQL

- Add support for setting the S3 and NFS backup schedule in a unified manner.
- Add version validation for MySQL sidecar image.
- Add support for backing up to NFS address in the **ip:/directory** format.
- Add support for validating MySQL images against MySQL version upon cluster creation.
- Add support for switching to the single-node mode when necessary.
- Support Chaos Mesh in E2E testing.
- Simplify cluster NFS backup with nfsbcp.
- Support MySQL configuration templates.
- Add support for customizing the MySQL version.
- Add capacity check when backing up to NFS.
- Add more status fields to the backup CRD.
- Update the version of Go to 1.17.
- Support helm installation of mysql-operator.

#### ECK

- Support the Hot-Warm-Cold architecture.
- Add support for deploying Kibana in the console.
- Add Elasticsearch Metrics Exporter.

#### Kafka

- Support Kafka v3.1 and v3.2.
- Add support for installing and managing Kafka and ZooKeeper nodes in the console.
- Support automatic cluster failover and recovery.
- Support cluster capacity and storage expansion.
- Add Kafka Metrics Exporter.
- Support secure access to the internal with TLS.
- Add support for managing users, topics, and ACLs in the console.
- Add support forrebalancing clusters in the console.

#### MongoDB

- Add service monitoring for MongoDB clusters.
- Enhance database account management.
- Support dynamic disk expansion.
- Optimize the elegant start and stop logic.
- Add support for displaying more cluster states.

#### OpenSearch

- Support installation and management of OpenSearch, Dashboard, and Logstash.
- Support OpenSearch v2.3.0.
- Support vertical expansion of the cluster computing and storage resources
- Support adjusting the number of cluster nodes.
- Support rolling updates upon configurations change.
- Support the keystore feature.
- Add OpenSearch Metrics Exporter.
- Support secure access with TLS.

#### PostgreSQL

- Support PostgreSQL kernel versions v12.12, v13.8, v14.5, and v15.0.
- Support the PostGIS and Citus plugins.
- Support horizontal and vertical scaling.
- Support semi-sync streaming replication.
- Support more metrics in the console.

#### Redis

- Add Proxy for Redis clusters.
- Support Redis Cluster failover and auto-recovery.
- Support resource monitoring.
- Add requests and limits.
- Support password and password-free settings for Redis clusters.
- Add support for dynamically reloading configurations upon Redis configuration changes.

## App Store

### Features

- Add support for managing helm releases created by running the native helm command.
- Add new options in OpenPitrix.

### Bug Fixes

- Fix a ks-controller-manager crash error caused by Helm controller NPE errors.
- Fix an issue that occurs while updating the traffic policy on the traffic monitoring page of an application.

## Network

### Enhancements & Updates

- Integrate OpenELB with KubeSphere for exposing LoadBalancer services.

### Bug Fixes

- Fix an issue where the gateway of a project is not deleted after the project is deleted.
- Fix an issue during gateway log query, where the log query connection is not terminated after the query is completed.
- Fix an issue where users fail to create routing rules in IPv6 and IPv4 dual-stack environments.
- Update the OpenELB check URL.

## Service Mesh

### Bug Fixes

- Resolve port conflicts of virtual services that use multiple protocols.

## KubeEdge Integration

### Features

- Add support for logging in to common cluster nodes and edge nodes from the KubeSphere web console.

### Enhancements & Upgrades

- Upgrade KubeEdge from v1.7.2 to v1.9.2.
- Remove EdgeWatcher as KubeEdge v1.9.2 provides similar functions.

### Bug Fixes

- Change the cluster module key from `kubeedge` to `edgeruntime`.

## License

### Features

- Add Cluster ID to the license control mechanism.
- Add license for the Observability Center.
- Add license for Spring Cloud.
- Add metrics and alert rules for the KubeSphere license, and notifications will be sent if the KubeSphere license is about to expire in less than 30 days.
- Add support for showing the license ID on the license page.

## Security

### Features

- Add Ingress to support HTTPS.

### Enhancements & Updates

- Enforce the password length to be at least 8 characters.
- Add the logic to check whether the username and password are the same.
- Disable password auto-filling.

## User Experience

### Features

- Display events for abnormal pods.
- Add in-memory cache in addition to the original Redis cache.
- Update the product name and logo to KubeSphere Enterprise.
- Add the End User License Agreements.
- Add support for more languages in the KubeSphere web console.
- Add the lifecycle management feature for containers.
- Add support for creating container environment variables in batches from secrets and configmaps.
- Add a message in the **Audit Log Search** dialog box, which prompts users to enable the audit logs feature.
- Add data units in the **Create Storage Class** dialog box.
- Add the cluster viewing permission to a user when the user adds a cluster.
- Add a time range selector on the **Traffic Monitoring** tab page.
- Add a prompt to remind users to select a language or artifact type when users are building images.
- Add support for changing the number of items displayed on each page of a table.
- Add support for batch stopping workloads.
- Add the creator annotation to ensure information displayed on the pod details page is consistent with other details pages.
- Add support for displaying the revision record when the workload type is `statefulsets` or `daemonsets`.
- Add a support page.
- Add support for duplicate name validation of containers and service names.

### Enhancements & Updates

- Add support for setting the default container registry.
- Support the shortcut dropdown of the relevant events of the pods.
- Support quick redirection to the destination page and settings of the rows displayed on the page.
- Add support for hiding external links in the console.
- Remove the limit to 100 of the cpu-percent of HPA.
- Optimize the service details area on the **Service Topology** page.
- Prevent passwords without uppercase letters set through the backend CLI.
- Optimize the UI texts of clusters, workspaces, and project deletion.
- Set the **Token** parameter on the webhook settings page as mandatory.
- Improve user experience in the web console of member clusters.
- Optimize the **Service Type** and **External Access** columns of the service list.
- Improve the registry verification API to resolve registry verification failures.

### Bug Fixes

- Fix an issue where statefulset creation fails when a volume is mounted to an init container.
- Fix an app installation failure, which occurs when users click buttons too fast.
- Fix an incorrect message displayed during workload creation, which indicates that resource requests have exceeded limits.
- Fix an issue where no data is displayed on the **Traffic Management** and **Tracing** tab pages in a multi-cluster project.
- Fix an issue where data in the service details area on the **Service Topology** page is not updated automatically.
- Fix an issue where existing settings are not displayed when a stateful service is being edited.
- Fix an issue where container probes are still displayed after they are deleted.
- Fix incorrect container names in the service details area on the **Service Topology** page.
- Fix the incorrect number of worker nodes on the **Cluster Nodes** page.
- Fix a workspace API error caused by an invalid cluster role.
- Fix an incorrect error message displayed when kubeconfig is updated for a second time.
- Fix an incorrect private key type in kubeconfig information.
- Fix an issue where the key is not display in LoadBalancer.
- Fix an issue where no prompt is displayed when users do not fill in key and value in the LoadBalancer access mode.
- Fix an issue where the update time of a service is incorrect on the details page.
- Fix the inaccurate prompt when users are adding an init container.
- Fix an issue where an error occurs while users enter Chinese characters in secret.
- Fix an issue where the total number of pages is incorrectly displayed.
- Fix an issue where the update time of an application is incorrect.
- Fix a 404 issue in Documentation.
- Fix an issue where the status of a cluster remains true when the cluster fails to join Federation.
- Fix an issue where traffic allocation fails in the canary release mode.
- Fix an issue where configurations do not take effect when users set the pod request to 0.
- Fix an issue where canary release goes wrong when multiple operating systems are selected.
- Fix an issue where configmap configurations cannot be saved while users are creating a workload.

## API Changes

- Upgrade the alerting API to v2beta1, and the v2alpha1 alerting API is deprecated.
- Expose kube-apiserver of the host cluster as a LoadBalancer service for member clusters to access.
- Provide RESTful APIs for ClusterTemplate.
- Provide template-related APIs.
- Change the KubeEdge proxy service to `http://edgeservice.kubeedge.svc/api/`.
- Change the patch type of `PatchWorkspaceTemplate` from `MergePatchType` to `JSONPatchType`.


v3.3.1 Release Notes

KubeSphere Enterprise v3.3.1 Release Notes

Release Notes

KubeSphere Enterprise Release Notes

KubeSphere Enterprise is a distributed multi-tenant cloud-native OS developed based on Kubernetes. Powered by feature-rich open-source KubeSphere, KubeSphere Enterprise delivers enhanced features in multi-cloud management, microservice goverance, application management, and other core business scenarios.