• Cloud-Native Container ProductsCloud-Native Container Products
    • KubeSphere Enterprisehot
    • KubeSphere Virtualizationhot
    • KubeSphere Enterprise HCI
  • Cloud-Native ServiceCloud-Native Service
    • KubeSphere Backuphot
    • KubeSphere Litenew
    • KubeSphere Inspectornew
  • Public Cloud Container ServicePublic Cloud Container Service
    • KubeSphere on AWS
    • KubeSphere on DigitalOcean

KubeSphere Enterprise v3.3.1 Release Notes

Starting from v3.3.1, the previous QingCloud KubeSphere Container Platform (QKCP) has been renamed to KubeSphere Enterprise.

Observability

Features

  • Add Whizard v0.5.0 as the Prometheus long-term storage for multi-cluster monitoring & alerting.
  • Add the Observability Center, which supports multi-cluster monitoring and alerting.
  • Introduce three new CRDs GlobalRuleGroups, ClusterRuleGroups, and RuleGroups for multi-cluster, cluster, and project alerting respectively. Alert rules are now defined in GlobalRuleGroups, ClusterRuleGroups or RuleGroups, which will be converted to groups in PrometheusRules.
  • Add a controller to sync PrometheusRules to the host cluster.
  • Add an admission webhook and controllers for RuleGroups, ClusterRuleGroups, and GlobalRuleGroups.
  • Built-in alert rules can be modified, enabled, disabled, and reset now.
  • Add support for alerting of KubeSphere core components.
  • Add support for cleaning up a cluster's notification settings when removing this cluster.
  • Add the Feishu notification channel.
  • Add support for silencing notifications.
  • Add support for changing the language of notifications.
  • Add support for adding a user account's email address to this user's notification settings.
  • Use OpenSearch as built-in log storage instead of Elasticsearch.
  • Add the OpenSearch curator.
  • Add the OpenSearch Dashboard.
  • Add OpenSearch as the log receiver.
  • Add support for searching logs from OpenSearch in the KubeSphere console.
  • Add process and thread monitoring metrics for containers.
  • Add disk monitoring metrics, for example, disk usage data.
  • Add support for importing Grafana templates to create a custom monitoring dashboard in a namespace.
  • Add support for defining data retention periods for container logs, Kubernetes event logs, and audit logs respectively.

Enhancements & Updates

  • Improve the cluster overview page.
  • Improve workspace and project monitoring.
  • Upgrade Prometheus to v2.39.1, Thanos to v0.29.5 (customized), and kube-state-metrics to v2.6.0.
  • Upgrade fluentbit-operator to v0.14.0.
  • Upgrade Notification Manager to v2.1.0.
  • Upgrade Alertmanager from v0.21.0 to v0.23.0.
  • Upgrade Grafana from v7.4.3 to v8.3.3.
  • Upgrade node-exporter from v0.18.1 to v1.3.1.
  • Upgrade Prometheus Operator from v0.43.2 to v0.55.1.
  • Upgrade kube-rbac-proxy from v0.8.0 to v0.11.0.
  • Upgrade configmap-reload from v0.3.0 to v0.5.0.
  • Upgrade kube-events from v0.3.0 to v0.4.0.
  • Upgrade Fluent Bit from v1.8.3 to v1.9.4.

Bug Fixes

  • Fix the style of log details in the toolbox.
  • Fix inaccurate time unit of the monitoring metrics.
  • Fix the "No Cluster Available" issue during log search.

DevOps

Features

  • Add the Continuous Deployment feature, which supports GitOps and uses Argo CD as the backend.
  • Optimize pipeline templates loaded from APIs.
  • Add the allowlist feature on the Basic Information page of a DevOps project.
  • Add the Events tab on the pipeline run record page.

Enhancements & Updates

  • Add support for editing the kubeconfig binding mode on the pipeline UI.
  • Add a message indicating that the S2I and B2I features currently do not support the containerd runtime.

Bug Fixes

  • Fix an issue where Jenkins updates are not synchronized in real time.
  • Fix the cron expression check failure of Jenkins.
  • Fix an issue in the Import Code Repository dialog box, where repositories and organizations are not completely displayed.
  • Fix an issue where users fail to check the CI/CD template.
  • Remove the Deprecated tag from the CI/CD template and replace kubernetesDeploy with kubeconfig binding at the deployment phase.
  • Fix an issue where pipeline parameters are not updated in time.
  • Fix the automatic cleaning behavior of NoSCMPipeline, and keep it consistent with the cleaning configuration of Jenkins.

Multi-Tenancy & Multi-Cluster

Features

  • Add support for grouping cluster by labels.
  • Add a limit on the number of labels that can be added to a cluster.
  • Add support for automatically renewing a cluster's kubeconfig.
  • Add metrics for cluster certificate expiration. Users will be notified when the kubeconfig is about to expire in 7 days or 24 hours.
  • Add support for removing the cluster warning message.
  • Add the name of current cluster to the kubesphere-config configmap.
  • Support cluster member management and cluster role management.

Bug Fixes

  • Set the ready status to false when a cluster fails to join the federation control plane.

Microservice

Features

  • Support Spring Cloud and add Spring Cloud Operator.
  • Support Spring Cloud configurations and Spring Cloud Microservice in Spring Cloud Operator.
  • Support Spring Cloud Gateway in Spring Cloud Operator.
  • Support Microservice Aggregated API in Spring Cloud Operator.
  • Add GatewayRouteConfig in Spring Cloud Operator.
  • Add the springcloud-controller helm chart.
  • Add kubesphere-spring-cloud-starter.

Storage

Features

  • Support PVC resizing, which can automatically expand PVCs when the remaining capacity is insufficient.
  • Support volume snapshot content management and volume snapshot class management.
  • Allow users to set authorization rules for storage classes so that storage classes can be used only in specific projects.
  • Provide usage data of each disk.

Bug Fixes

  • Set hostpath as a required option when users are mounting volumes.
  • Update storageClass-accessor, so that deleting storage resources no longer requires accessor validation.

Authentication & Authorization

Features

  • Add support for manually disabling and enabling users.

Enhancements & Updates

  • Check user permissions to clusters when users are creating or updating workspace templates.

Bug Fixes

  • Add support for filtering workspace roles using the LabelSelector.
  • Add support for customizing or randomly setting an initial admin password.
  • Delete annotations in role-template-manage-users, role-template-view-members and role-template-manage-roles.
  • Fix an issue where cluster-admin cannot view and manage the configmap, secret, and service account.
  • Delete role workspace-manager.
  • Add role platform-self-provisioner.
  • Delete role users-manager.
  • Block role-template-manage-groups.

DMP

Features

DMP Framework

  • Refactor monitoring by adding application monitoring information.

ClickHouse

  • Support ClickHouse kernel version 21.1.3.32.
  • Support more ClickHouse metrics.
  • Support horizontal and vertical expansion.
  • Support SVC-based cluster load balancing.
  • Support ClickHouse HA.
  • Support more cluster states.
  • Support disk resizing with the QingCloud CSI.
  • Add support for automatically provisioning and managing ZooKeeper clusters.

MySQL

  • Add support for setting the S3 and NFS backup schedule in a unified manner.
  • Add version validation for MySQL sidecar image.
  • Add support for backing up to NFS address in the ip:/directory format.
  • Add support for validating MySQL images against MySQL version upon cluster creation.
  • Add support for switching to the single-node mode when necessary.
  • Support Chaos Mesh in E2E testing.
  • Simplify cluster NFS backup with nfsbcp.
  • Support MySQL configuration templates.
  • Add support for customizing the MySQL version.
  • Add capacity check when backing up to NFS.
  • Add more status fields to the backup CRD.
  • Update the version of Go to 1.17.
  • Support helm installation of mysql-operator.

ECK

  • Support the Hot-Warm-Cold architecture.
  • Add support for deploying Kibana in the console.
  • Add Elasticsearch Metrics Exporter.

Kafka

  • Support Kafka v3.1 and v3.2.
  • Add support for installing and managing Kafka and ZooKeeper nodes in the console.
  • Support automatic cluster failover and recovery.
  • Support cluster capacity and storage expansion.
  • Add Kafka Metrics Exporter.
  • Support secure access to the internal with TLS.
  • Add support for managing users, topics, and ACLs in the console.
  • Add support forrebalancing clusters in the console.

MongoDB

  • Add service monitoring for MongoDB clusters.
  • Enhance database account management.
  • Support dynamic disk expansion.
  • Optimize the elegant start and stop logic.
  • Add support for displaying more cluster states.

OpenSearch

  • Support installation and management of OpenSearch, Dashboard, and Logstash.
  • Support OpenSearch v2.3.0.
  • Support vertical expansion of the cluster computing and storage resources
  • Support adjusting the number of cluster nodes.
  • Support rolling updates upon configurations change.
  • Support the keystore feature.
  • Add OpenSearch Metrics Exporter.
  • Support secure access with TLS.

PostgreSQL

  • Support PostgreSQL kernel versions v12.12, v13.8, v14.5, and v15.0.
  • Support the PostGIS and Citus plugins.
  • Support horizontal and vertical scaling.
  • Support semi-sync streaming replication.
  • Support more metrics in the console.

Redis

  • Add Proxy for Redis clusters.
  • Support Redis Cluster failover and auto-recovery.
  • Support resource monitoring.
  • Add requests and limits.
  • Support password and password-free settings for Redis clusters.
  • Add support for dynamically reloading configurations upon Redis configuration changes.

App Store

Features

  • Add support for managing helm releases created by running the native helm command.
  • Add new options in OpenPitrix.

Bug Fixes

  • Fix a ks-controller-manager crash error caused by Helm controller NPE errors.
  • Fix an issue that occurs while updating the traffic policy on the traffic monitoring page of an application.

Network

Enhancements & Updates

  • Integrate OpenELB with KubeSphere for exposing LoadBalancer services.

Bug Fixes

  • Fix an issue where the gateway of a project is not deleted after the project is deleted.
  • Fix an issue during gateway log query, where the log query connection is not terminated after the query is completed.
  • Fix an issue where users fail to create routing rules in IPv6 and IPv4 dual-stack environments.
  • Update the OpenELB check URL.

Service Mesh

Bug Fixes

  • Resolve port conflicts of virtual services that use multiple protocols.

KubeEdge Integration

Features

  • Add support for logging in to common cluster nodes and edge nodes from the KubeSphere web console.

Enhancements & Upgrades

  • Upgrade KubeEdge from v1.7.2 to v1.9.2.
  • Remove EdgeWatcher as KubeEdge v1.9.2 provides similar functions.

Bug Fixes

  • Change the cluster module key from kubeedge to edgeruntime.

License

Features

  • Add Cluster ID to the license control mechanism.
  • Add license for the Observability Center.
  • Add license for Spring Cloud.
  • Add metrics and alert rules for the KubeSphere license, and notifications will be sent if the KubeSphere license is about to expire in less than 30 days.
  • Add support for showing the license ID on the license page.

Security

Features

  • Add Ingress to support HTTPS.

Enhancements & Updates

  • Enforce the password length to be at least 8 characters.
  • Add the logic to check whether the username and password are the same.
  • Disable password auto-filling.

User Experience

Features

  • Display events for abnormal pods.
  • Add in-memory cache in addition to the original Redis cache.
  • Update the product name and logo to KubeSphere Enterprise.
  • Add the End User License Agreements.
  • Add support for more languages in the KubeSphere web console.
  • Add the lifecycle management feature for containers.
  • Add support for creating container environment variables in batches from secrets and configmaps.
  • Add a message in the Audit Log Search dialog box, which prompts users to enable the audit logs feature.
  • Add data units in the Create Storage Class dialog box.
  • Add the cluster viewing permission to a user when the user adds a cluster.
  • Add a time range selector on the Traffic Monitoring tab page.
  • Add a prompt to remind users to select a language or artifact type when users are building images.
  • Add support for changing the number of items displayed on each page of a table.
  • Add support for batch stopping workloads.
  • Add the creator annotation to ensure information displayed on the pod details page is consistent with other details pages.
  • Add support for displaying the revision record when the workload type is statefulsets or daemonsets.
  • Add a support page.
  • Add support for duplicate name validation of containers and service names.

Enhancements & Updates

  • Add support for setting the default container registry.
  • Support the shortcut dropdown of the relevant events of the pods.
  • Support quick redirection to the destination page and settings of the rows displayed on the page.
  • Add support for hiding external links in the console.
  • Remove the limit to 100 of the cpu-percent of HPA.
  • Optimize the service details area on the Service Topology page.
  • Prevent passwords without uppercase letters set through the backend CLI.
  • Optimize the UI texts of clusters, workspaces, and project deletion.
  • Set the Token parameter on the webhook settings page as mandatory.
  • Improve user experience in the web console of member clusters.
  • Optimize the Service Type and External Access columns of the service list.
  • Improve the registry verification API to resolve registry verification failures.

Bug Fixes

  • Fix an issue where statefulset creation fails when a volume is mounted to an init container.
  • Fix an app installation failure, which occurs when users click buttons too fast.
  • Fix an incorrect message displayed during workload creation, which indicates that resource requests have exceeded limits.
  • Fix an issue where no data is displayed on the Traffic Management and Tracing tab pages in a multi-cluster project.
  • Fix an issue where data in the service details area on the Service Topology page is not updated automatically.
  • Fix an issue where existing settings are not displayed when a stateful service is being edited.
  • Fix an issue where container probes are still displayed after they are deleted.
  • Fix incorrect container names in the service details area on the Service Topology page.
  • Fix the incorrect number of worker nodes on the Cluster Nodes page.
  • Fix a workspace API error caused by an invalid cluster role.
  • Fix an incorrect error message displayed when kubeconfig is updated for a second time.
  • Fix an incorrect private key type in kubeconfig information.
  • Fix an issue where the key is not display in LoadBalancer.
  • Fix an issue where no prompt is displayed when users do not fill in key and value in the LoadBalancer access mode.
  • Fix an issue where the update time of a service is incorrect on the details page.
  • Fix the inaccurate prompt when users are adding an init container.
  • Fix an issue where an error occurs while users enter Chinese characters in secret.
  • Fix an issue where the total number of pages is incorrectly displayed.
  • Fix an issue where the update time of an application is incorrect.
  • Fix a 404 issue in Documentation.
  • Fix an issue where the status of a cluster remains true when the cluster fails to join Federation.
  • Fix an issue where traffic allocation fails in the canary release mode.
  • Fix an issue where configurations do not take effect when users set the pod request to 0.
  • Fix an issue where canary release goes wrong when multiple operating systems are selected.
  • Fix an issue where configmap configurations cannot be saved while users are creating a workload.

API Changes

  • Upgrade the alerting API to v2beta1, and the v2alpha1 alerting API is deprecated.
  • Expose kube-apiserver of the host cluster as a LoadBalancer service for member clusters to access.
  • Provide RESTful APIs for ClusterTemplate.
  • Provide template-related APIs.
  • Change the KubeEdge proxy service to http://edgeservice.kubeedge.svc/api/.
  • Change the patch type of PatchWorkspaceTemplate from MergePatchType to JSONPatchType.

KubeSphere ®️ © QingCloud Technologies 2022