KubeSphere Enterprise v4.1.0 Release Notes

Release Notes

KubeSphere Core
Observability
Platform Management
Database and Middleware
- RadonDB DMP v2.1.0
DevOps
- DevOps v1.1.0
Networking
- Gateway v1.0.0
- Network v1.0.0
Security
- Gatekeeper v1.0.0
Storage
- Storage v1.0.0

KubeSphere Core

KubeSphere Enterprise (KSE) v4.1.0 is a brand new cloud-native operating system developed by the KubeSphere team at QingCloud. It is the first full-featured version of KubeSphere Enterprise after the architectural revolution.

Based on the new cloud-native, scalable, and open architecture of KubeSphere LuBan, KubeSphere Enterprise v4.1.0 decouples the full product functionalities of KubeSphere Enterprise v3.x and refactors and upgrades them based on a pluggable architecture specification. As a result, each KubeSphere extension can be independently released and iterated. Users can install extensions based on their desired product capabilities, maintaining a lightweight and flexible platform, and effortlessly customizing their own exclusive operating system tailored to their needs.

KubeSphere Enterprise v4.1.0 comes with a rich and versatile KubeSphere Marketplace. The released extensions have undergone strict review, ensuring high quality and controllability, and covers all aspects of cloud-native businesses. Users can install extensions with one click, and manage the full lifecycle of extensions through the Extensions Center. Additionally, enterprises or individual developers can follow the development specifications of KubeSphere LuBan to publish their own software and services on the KubeSphere Marketplace for distribution and commercial sales, enriching the diverse cloud-native application ecosystem.

Features

Refactor based on the new microkernel architecture of KubeSphere LuBan.
Introduce the KubeSphere Marketplace as a built-in feature.
Support for managing extensions through the Extensions Center.
UI and API can be extended.
Support for one-click import of member clusters via kubeconfig.
Support for KubeSphere Service Accounts.
Integrated TOTP-based two-factor authentication.
Support for dynamic extension of the Resource API.
Support for adding clusters, workspaces, and projects to quick access.
Enabled file upload and download via container terminal.
Adapted to cloud-native gateways (Kubernetes Ingress API) from different vendors.

Enhancements

Support for selecting all clusters when creating a workspace.
Optimization of web kubectl, supporting dynamic recycling of pods and fuzzy search when switching clusters.
Optimization of node list, changing the default sorting to ascending order.
Only allow trusted OAuth clients to verify user identity directly using username and password.
Streamline the Agent components deployed in member clusters.
Split some configurations in KubeSphere Config as independent configuration items.
Adjust the search result of container images to sort in reverse chronological order.
Support for editing user aliases.
Display scheduling status in the cluster list.
Support binaryData data display in ConfigMap details.
Refactor the Workbench page.

Bug Fixes

Fix the issue of the node terminal displaying "connecting" indefinitely.
Fix the potential issue of unauthorized access to resources in the workspace.
Fix the potential issue of unauthorized access to cluster authorization APIs in the workspace.
Fix the issue of abnormal session logout due to incorrect configuration.
Fix the issue of exceptions when adding image service information to pull images from a specified repository.
Fix the issue of missing ownerReferences when editing Secrets.
Fix the issue of white screen and incorrect page redirection during the initial login.
Fix the scrolling issue with checkboxes in Windows environment.
Fix the problem where the cluster management entry couldn't be found when logged in as cluster-admin.

API Updates

API Removal

The following APIs will be removed in v4.1:

Multi-cluster

The multi-cluster proxy request API /API_PREFIX/clusters/{cluster}/API_GROUP/API_VERSION/... has been removed. Please use the new multi-cluster proxy request path rule /clusters/{cluster}/API_PREFIX/API_GROUP/API_VERSION/... instead.

Access Control

The iam.kubesphere.io/v1alpha2 API version has been removed. Please use the iam.kubesphere.io/v1beta1 API version instead.
Significant changes in iam.kubesphere.io/v1beta1: The API Group for Role, RoleBinding, ClusterRole, and ClusterRoleBinding resources has changed from rbac.authorization.k8s.io to iam.kubesphere.io.

Multi-tenancy

Partial APIs in tenant.kubesphere.io/v1alpha1 and tenant.kubesphere.io/v1alpha2 API versions have been removed. Please use the tenant.kubesphere.io/v1beta1 API version instead.
Significant changes in tenant.kubesphere.io/v1beta1: spec.networkIsolation in Workspace has been removed.

kubectl

The /resources.kubesphere.io/v1alpha2/users/{user}/kubectl interface has been removed. Terminal-related operations no longer need to call this interface.
The API path for the user web kubectl terminal has been adjust from /kapis/terminal.kubesphere.io/v1alpha2/namespaces/{namespace}/pods/{pod}/exec to /kapis/terminal.kubesphere.io/v1alpha2/users/{user}/kubectl.

Gateway

The gateway.kubesphere.io/v1alpha1 API version has been removed.

The API for querying related gateways of the Ingress configuration has been adjust to /kapis/gateway.kubesphere.io/v1alpha2/namespaces/{namespace}/availableingressclassscopes.

API Deprecations

The following APIs have been marked as deprecated and will be removed in future versions:

Cluster validation API
Config configz API
OAuth token review API
Operations job rerun API
Resources v1alpha2 API
Resources v1alpha3 API
Tenant v1alpha3 API
Legacy version API

Known Issues

LDAP Identity Provider will be supported in future versions.
Department management in workspaces will be supported in future versions.
KEDA will be supported in future versions.

Misc

Remove all language options except English and Simplified Chinese by default.
Remove content related to system components.

Observability

The observability-related features in KubeSphere Enterprise v3.x, such as monitoring, alerting, logging, auditing, events, notification, and events alerting, have undergone significant refactoring and optimization in v4.1.0, gradually unified in the WhizardTelemetry Observability Platform and released v1.0.0 along with KubeSphere Enterprise v4.1.0. In addition to optimizations and refactoring in directions such as architecture, APIs, and technology stacks, reducing unnecessary workloads is also an important goal of the WhizardTelemetry Observability Platform v1.0.0:

The observability-related features in KubeSphere Enterprise v3.x have been split into 10 extensions conforming to the KubeSphere LuBan architecture, including WhizardTelemetry Platform Service, WhizardTelemetry Monitoring, WhizardTelemetry Alerting, WhizardTelemetry Logging, WhizardTelemetry Events, WhizardTelemetry Auditing, WhizardTelemetry Notification, WhizardTelemetry Events Alerting, WhizardTelemetry Data Pipeline, and OpenSearch Distributed Distributed Search and Analytics Engine. These extensions collectively form the WhizardTelemetry Observability Platform. Users can install the required extensions according to their needs, without having to install unnecessary components by default as in KubeSphere Enterprise v3.5.0 and earlier versions.
While maintaining the original style of the KubeSphere monitoring API, there have been significant adjustments and changes to the monitoring API.
Significant architectural and API optimizations and adjustments have been made to alerting and notification, including:
- Optimization of processes in multi-cluster mode, especially significant reduction of processes when the Observability Center is enabled.
- Make member clusters more lightweight in multi-cluster mode, where member clusters do not need to deploy workloads for alerting when the Observability Center is enabled.
- Alertmanager and Notification Manager, originally required to be deployed to each cluster, are now only deployed to the host cluster. Additionally, an Alertmanager Proxy is added to the host cluster to receive alerting from member clusters and forward them to Alertmanager.
Significant architectural refactoring and optimization have been done for functions such as logging, auditing, events, and notification history, achieving true multi-cluster architecture:
- The Agent Fluent Bit used for collecting information for logging, auditing, events, and notification history has been replaced by Vector Agent, and Vector Aggregator has been added to deployed to the host cluster to collect notification history and other information.
- OpenSearch, originally deployed to each cluster, can now be shared by multiple clusters belonging to the same region or organization, eliminating the need for installation on each cluster. Different clusters will create indexes with the cluster name as a prefix.
- The kube-auditing-webhook used for receiving audit data has been removed, and audit data will be collected by Vector Agent after being stored.
- Kube-events controller, ruler, and CRDs used for managing native Kubernetes events have been removed, leaving only kube-events-exporter for exporting Kubernetes events.
- The archiving of Kubernetes and KubeSphere audit information has been unified to be collected by Vector Agent from files written to disk, removing the previous Webhook.
- The alerting function for events and auditing has been merged into WhizardTelemetry Events Alerting, which will support log alerting in this extension in the future. The alerting rules for events and auditing have also been merged into a single CRD ClusterRuleGroup.logging.whizard.io.

WhizardTelemetry Platform Service v1.0.0

The WhizardTelemetry Platform Service is a newly added service formed by extracting observable-related features from the original KubeSphere APIServer. It serves as the shared APIServer for various observable services in the WhizardTelemetry Observability Platform, providing a common backend platform service for all observability functions. Currently, it provides APIs for monitoring, logging, auditing, events, notifications, and other services.

Features

Integrate KubeSphere authentication, supporting user permission verification.
Monitoring API supports loading PromQL query expressions via template files.
Monitoring API supports custom component queries.

Enhancements

Improve query performance of monitoring API.
Improve query performance of logging API.
Improve query performance of auditing API.
Improve query performance of events API.
Improve query performance of notification history API.

Deprecations

The monitoring API monitoring.kubesphere.io/v1alpha3 was deprecated in KubeSphere Enterprise v3.5.0 and officially removed in KubeSphere Enterprise v4.1.0.
The logging, auditing, and events APIs tenant.kubesphere.io/v1alpha2 were officially removed in KubeSphere Enterprise v4.1.0.

API Updates

The monitoring API has been upgrade to version monitoring.kubesphere.io/v1beta1. For more API details, refer to the WhizardTelemetry Monitoring API Reference and Swagger.
The logging, auditing, events, and notification history APIs have been upgrade to version logging.kubesphere.io/v1beta2. For more API details, refer to the WhizardTelemetry API Reference and Swagger.

WhizardTelemetry Monitoring v1.0.0

WhizardTelemetry Monitoring is an extension in the WhizardTelemetry Observability Platform that provides monitoring functionality. It includes the Whizard Observability Center, which offers cloud-native resource monitoring capabilities from a multi-tenant perspective, and includes real-time and historical data display of core monitoring metrics for multi-cluster, nodes, workloads, GPUs, and Kubernetes control planes.

This extension deploys and manages the following components:

Whizard Prometheus Long-term Storage
Whizard is an enterprise-level multi-cloud and multi-cluster monitoring and alerting product that addresses long-term storage issues of monitoring and alerts based on metrics in the observability field.
Kube-Prometheus-Stack
Kube-Prometheus-Stack monitors Kubernetes clusters and applications running on them using Prometheus. It includes components such as Prometheus Operator, kube-state-metrics, node-exporter, and configuration manifest for collecting kubernetes component metrics and related Prometheus Rules.
Calico Exporter (deployed within Kube-Prometheus-Stack's node-exporter, can be enabled separately)
Calico Exporter collects performance metrics for Calico networks.
Process-exporter (deployed within Kube-Prometheus-Stack's node-exporter, can be enabled separately)
Process-exporter collects performance metrics for processes.
DCGM-Exporter
DCGM-Exporter collects performance and health metrics for Nvidia GPUs.
Whizard Monitoring Helper
A utility tool for deploying WhizardTelemetry Monitoring.

Features

Support exposing query UI and querying external data sources with Whizard Gateway.
Support authentication and authorization for accessing external data sources with Whizard.
Support component configuration at the Services level with Whizard, allowing for differentiated configurations.
Optimize internal TLS configuration to improve query performance with Whizard.
Refer to the Whizard v0.10.0 Release for more features and optimizations.
Kube-Prometheus-Stack has full compatibility with the community project kube-prometheus.
Integrate Calico Exporter and Process-exporter into the node-exporter of Kube-Prometheus-Stack.
Optimize default metric configuration and support collecting more GPU metrics with DCGM-Exporter.

Enhancements

Optimize recording rules configuration in Kube-Prometheus-Stack to synchronize with the latest community updates.
Optimize resource quotas for components in Kube-Prometheus-Stack, and remove unused metrics to improve performance.

Bug Fixes

Fix the issue where custom rules on the rules group page of the cluster/project were not automatically suggested when the observability center is enabled.

API Updates

Integrate the monitoring API into the WhizardTelemetry Platform Service. Refer to the WhizardTelemetry Platform Service Changelog for API changes.

Misc

Upgrade Whizard to v0.10.0.
Upgrade Prometheus Operator to v0.68.0.
Upgrade Prometheus to v2.50.1.
Upgrade kube-state-metrics to v2.8.2.
Upgrade node-exporter to v1.6.0.
Upgrade DCGM-Exporter to v3.4.0.

WhizardTelemetry Alerting v1.0.0

WhizardTelemetry Alerting is an extension in the WhizardTelemetry Observability Platform that manages alerts based on monitoring metrics. It provides functions such as multi-cluster rule group management, evaluation, and alerting.

This extensions deploys and manages the following components:

apiserver: Provides APIs related to rule groups and alerts.
controller-manager: Provides functions such as synchronized management of rule groups.
ruler: Responsible for rule evaluation and alerting for rule groups.

Compared to KubeSphere Enterprise v3.5.0, this extension also significantly optimizes the process and lightweighting of multi-cluster alerts from an architectural design perspective.

Enhancements

Optimize alerting process, significantly reducing alerting process, especially when the Observability Center is enabled.
Reduce alerting workload for member clusters, achieving lightweighting of alerting workload for member clusters when the Observability Center is enabled.

Bug Fixes

Fix the issue where zero timestamps briefly appear during rule checks.
Fix the issue that no data display and pagination abnormalities when querying alerts using multiple filter conditions on the Alerting page.

API Updates

Compared to KubeSphere Enterprise v3.5.0, the API updates mainly involve changes to the request paths:

For cluster-level and project-level rule groups and alerts, the API path prefix is changed from [apis|kapis]/clusters/{cluster}/alerting.kubesphere.io/v2beta1/ to /proxy/alerting.kubesphere.io/v2beta1/clusters/{cluster}/.
For global-level rule groups and alerts, the API path prefix is changed from [apis/kapis]/alerting.kubesphere.io/v2beta1/ to /proxy/alerting.kubesphere.io/v2beta1/.
For built-in rule groups, access is made through the cluster-level rule group API path and a request parameter builtin=true when the Observability Center is not enabled. For example, requesting /proxy/alerting.kubesphere.io/v2beta1/clusters/{cluster}/[clusterrulegroups|clusteralerts]?builtin=true allows access to built-in rule groups and their alerts respectively.

The data structure of request body and response remain unchanged.

For more details, please refer to the API documentation.

Misc

The process-exporter-rules rule group, as a built-in rule group, supports management through global rule groups when the Observability Center is enabled.

WhizardTelemetry Logging v1.0.0

WhizardTelemetry Logging is an extension in the WhizardTelemetry Observability Platform used for log collection, processing, storage, and querying.

This extension deploys and manages the following components:

logsidecar-injector is used for collecting logs written by containers in Pods to disk (different from logs output to stdout by regular containers).
K8s log collection and transformation
Default OpenSearch sink

Features

Add Vector Agent for default log collection to disk
Change the deployment of OpenSearch from being deployed per cluster to being shared across multiple clusters, with different clusters creating indexes prefixed with the cluster name
Support outputting logs from some clusters (e.g., Zone A or Department A) to one OpenSearch instance, and logs from other clusters (e.g., Zone B or Department B) to different OpenSearch instances. After configuration, logs from all clusters can be queried in the same log console. For specific details, please refer to the README.

Bug Fixes

Fix the issue of slow frontend page access when querying logs.

Deprecations

The filebeat previously used for collecting logs to disk has been deprecated and will be removed in future versions.

WhizardTelemetry Events v1.0.0

WhizardTelemetry Event Management is an extension in the WhizardTelemetry Observability Platform used to export Kubernetes native events. This extension deploys and manages kube-events-exporter. It primarily:

Collects Kubernetes native events and export them to stdout.
Vector Agent collects disk-based Kubernetes event logs, performs format conversion, and sends them to user-specified receivers such as OpenSearch.
Supports querying Kubernetes event logs for each cluster on the UI.

Features

Add kube-events-exporter for exporting Kubernetes native events.
Transition from previous webhook-based reception of Kubernetes events to collecting Kubernetes events by Vector Agent, outputting them to stdout through kube-events-exporter, and then archiving or querying them through OpenSearch.

Deprecations

Deprecate and remove CRDs, Ruler, and Controller defined by the original kube-events project.

WhizardTelemetry Auditing v1.0.0

WhizardTelemetry Auditing is an extension in the WhizardTelemetry Observability Platform used for archiving and querying audit information for Kubernetes and KubeSphere. This extension primarily:

Vector Agent collects disk-based Kubernetes and KubeSphere audit logs and performs format conversion.
Sends the converted audit logs to specified receivers such as OpenSearch.
Supports querying audit logs for each cluster on the UI.

Enhancements

The collection of audit logs transitions from the previous webhook-based reception to Vector Agent collecting disk-based audit log files.

Deprecations

The original webhook for receiving audit logs has been deprecated and removed.
The CRD Rule.auditing.kubesphere.io for defining audit alert rules has been deprecated and removed.

WhizardTelemetry Notification v2.5.1

WhizardTelemetry Notification is an extension in the WhizardTelemetry Observability Platform that provides notification functionality, supporting multiple channels such as email, Slack, WeChat Work, DingTalk, Feishu, Webhook, etc.

This extension deploys and manages the following components:

Notification Manager
Notification History
Alertmanager
Alertmanager Proxy

Features

Support configuring receivers for notification history.

Enhancements

Optimize notification history collection method by removing the notification adaptor component.
Support for displaying receiver names in notifications.
Support for adding annotations and labels to the notification manager deployment.

Bug Fixes

Fix the issue where regex matching would match all alerts.
Fix the issue where the page would turn white after editing notification subscription conditions and selecting "Contains" as the filter condition.

Deprecations

Remove CRDs in version v2beta1.
Remove the notification adaptor component.

Misc

Upgrade Notification Manager to v2.5.1.
Upgrade Alertmanager to v0.26.0.

WhizardTelemetry Events Alerting v1.0.0

WhizardTelemetry Events Alerting is an extension in the WhizardTelemetry Observability Platform that provides events alerting functionality. It allows defining alert rules for Kubernetes native events and Kubernetes/KubeSphere audit events, evaluating incoming event data, and sending alerts to specified receivers such as HTTP endpoints.

This is the first official version of this extension, compatible with KubeSphere Enterprise v4.1.0.

Features

Add a cluster field to the issued alerts to distinguish which cluster the event alerts are coming from.
Support for sending messages to Webhook or other destinations by configuring receivers, such as Alertmanager Proxy deployed on the host cluster: http://< host node ip >:31093/api/v1/alerts

Enhancements

Merge the functionality of evaluating Kubernetes native events and Kubernetes/KubeSphere audit events based on alert rules from the original kube-events and kube-auditing projects into WhizardTelemetry Events Alerting.
Merge the respective alert rules from the original kube-events and kube-auditing projects into a single CRD ClusterRuleGroup.logging.whizard.io.

OpenSearch Distributed Search and Analytics Engine v2.11.1

OpenSearch Distributed Search and Analytics Engine is an integrated distributed search and analytics engine within the WhizardTelemetry Observability Platform. It serves as an extension for storing, retrieving, and analyzing observable data such as logs, audits, events, and notification histories.

This extension deploys and manages the following components:

Master nodes of OpenSearch
Data nodes of OpenSearch
OpenSearch Dashboard
OpenSearch Curator (used for periodic cleaning of expired data)

Enhancements

Adjust the Service type of OpenSearch data nodes to NodePort (port 30920).
Reduce CPU and memory requests for OpenSearch master nodes and data nodes.
Adjust the index cleaning rules for OpenSearch Curator.

Misc

Upgrade OpenSearch to v2.11.1.
Upgrade OpenSearch Dashboard to v2.11.1.

WhizardTelemetry Data Pipeline v1.0.0

WhizardTelemetry Data Pipeline is an extension in the WhizardTelemetry Observability Platform that provides capabilities for data collection, transformation, and routing.

Features

Replace the Fluent Bit previously used for data collection of logging, auditing, events, and notification history with Vector Agent.
Introduce Vector Aggregator for collecting notification history and other information on host clusters.
Add the vector-config sidecar container to listen for secrets storing Vector configuration and automatically generate Vector configuration files.
Customize the Vector Helm Chart to allow storing Vector configuration in a secret, replacing the previous practice of storing Vector configuration containing sensitive information in a ConfigMap.

Deprecations

Deprecate and remove the Fluent Bit and FluentBit Operator previously used for data collection.

Platform Management

App Store Management v2.0.0

App Store Management is a multi-cloud application management platform based on OpenPitrix, used for uploading, reviewing, and managing different types of applications in multi-cloud environments.

In v2.0.0, the synchronization performance of repository applications has been improved, and creating application templates using YAML files has been supported.

Features

Support for application uploading, review, release and unrelease.
Support for creating application templates using YAML files.
Support for external S3 object storage.
Support for global application repository configuration.
Support for more granular permission configurations, including viewing, creating, deleting, and overall management of applications, application versions, and application instances.

Enhancements

Improve product interaction, delineating the functional boundaries between App Store and App Store Management.
Remove built-in open-source repositories and open-source application templates.
Improve synchronization performance of repository applications.

Bug Fixes

Fix the issue where CRD could not be used immediately after installing Helm applications.

Deprecations

Remove APIs of the openpitrix.io/v1 series.
Remove APIs of the manifests.application.kubesphere.io series.

API Changes

Add APIs for creating YAML applications.
Utilize unified pagination and filtering conditions queries provided by the KubeSphere platform.

Service Mesh v1.0.0

Service Mesh is a powerful microservice governance and visualization management tool. It provides three types of gray release strategies, including blue-green deployment, canary release, and traffic mirroring, as well as observability capabilities such as traffic monitoring and distributed tracing.

Features

Support for more granular permission configurations, including service mesh management and viewing.

Enhancements

Improved product interaction by changing the entry points for "Composed Applications" and "Gray Releases."

Misc

Upgrade Istio from v1.14.6 to v1.16.5.
Upgrade Kiali from v1.50 to v1.59.
Upgrade Jaeger from v1.29 to v1.35.

Spring Cloud v1.0.0

Spring Cloud is an extension that provides functionalities for microservices, microservice configuration, and microservice gateway.

Features

Support for more granular permission configuration, including Spring Cloud management and viewing.

Enhancements

Reduce unnecessary logging in spring-cloud-controller.
Provide friendly reminders when enabling Spring Cloud functionality in existing projects.

Misc

Upgrade spring-cloud-controller from v0.1.0 to v0.1.1.

Multi-Cluster Agent Connection v1.0.0

Multi-Cluster Agent Connection is a tool for network connection between clusters through the agent. If the host cluster cannot access the member cluster directly, you can expose the proxy service address of the host cluster. This enables the member cluster to connect to the host cluster through the agent.

This extension refactors the multi-cluster proxy connection module from KubeSphere Enterprise v3.5.0 using the new microkernel architecture KubeSphere LuBan, while maintaining the functionality unchanged.

Application Management for Cluster Federation v1.0.0

Application Management for Cluster Federation is an extension aimed at simplifying the management of applications across federated Kubernetes clusters. It offers a unified interface for deploying, updating, and managing applications across federated clusters with flexible configuration options to meet diverse needs.

This extension refactors the multi-cluster project module from KubeSphere Enterprise v3.5.0 using the new microkernel architecture KubeSphere LuBan, while maintaining the functionality unchanged.

KubeEdge Edge Computing Framework v1.0.0

KubeEdge Edge Computing Framework is an extension that extends native containerized application orchestration and device management to hosts at the Edge.

This extension deploys and manages the following components:

cloudcore: Supports cloud-side access functionality.
frontend: Supports frontend interface functionality.

Misc

This extension currently only supports host clusters.

Metrics Server v0.7.0

Metrics Server is a scalable and efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.

This extension refactors metrics-server based on the new microkernel architecture KubeSphere LuBan. The code, images, and others remain unchanged.

Misc

Upgrade metrics-server from v0.4.2 to v0.7.0.

Database and Middleware

RadonDB DMP v2.1.0

The RadonDB DMP extension provides a unified visual management interface for managing and operating databases and middleware. It supports the full lifecycle management of database and middleware instances. It includes features such as manual backup, automated backup, recovery from backups, and display of monitoring metrics with alerting capabilities (observability extensions should be installed). It supports the following databases and middleware: MySQL, PostgreSQL, MongoDB, Redis, OpenSearch, Kafka, and RabbitMQ.

Features

Support for modifying database instances by editing YAML.
Support for separately configuring S3 addresses for database backup.

Bug Fixes

Fix the issue where directly modifying database configurations through the command line would reset the configurations.
Fix the issue where Logstash is not associated with roles, resulting in the inability to write data to OpenSearch.
Fix the issue where querying by conditions in OpenSearch monitoring does not receive a response.
Adjust PostgreSQL's resource specifications to avoid pod restarts due to insufficient memory.

Deprecations

Deprecate manifests.application.kubesphere.io.

Known Issues

Manual fixes are required to recover from MySQL backups.
Manual fixes are required to recover from PostgreSQL backups.

DevOps

DevOps v1.1.0

KubeSphere DevOps System is designed specifically for CI/CD workflows in Kubernetes. It provides an all-in-one solution to help development and operations teams build, test, and deploy applications to Kubernetes in a very simple way. It features functionalities such as plugin management, code dependency caching, code quality analysis, and pipeline logging. It is compatible with third-party private image registry (such as Harbor) and code repositories (such as GitLab/GitHub/SVN/BitBucket). It offers users a comprehensive and visual CI/CD pipeline, delivering an excellent user experience. Additionally, its strong compatibility makes it highly useful in offline environments.

Compared to KubeSphere Enterprise v3.5.0, this extension not only fixes known issues, enhances functionalities, and improves usability, but also reduces its reliance on KubeSphere Core. This makes the DevOps more independent and provides more flexibility in version management.

Features

Automatically identify the runtime environment during deployment to adapt the Agent image automatically.
Support viewing details of canceled pipelines.
Adjust DevOps RoleTemplates to adapt to KubeSphere LuBan IAM, providing more flexible and convenient DevOps permission management.

Enhancements

Support log viewing functionality on the pipeline details page.
Improve devops-controller log output for clearer information.
Adjust the description of branch cleanup in multi-branch pipelines.

Bug Fixes

Fix the issue where devops-controller fails to start due to empty cloneOptions.time in pipelines.
Fix the issue where parameters defined in pipelines are not passed to the Jenkins service.
Fix the error that occurs when the "Print Message" step in a pipeline contains double quotes.
Fix the issue where attachments fail to download in multi-branch pipelines.
Fix the issue where replaying a pipeline fails.
Fix the issue where filtering DevOps project aliases is invalid.

Known Issues

Image builders (S2I, B2I) functionality is not available in this version.
kubeconfig type credentials are not available in this version.

API Updates

DevOps-related APIs in ks-core have been moved to this extension. The path kapis/tenant.kubesphere.io/v1alpha2 has been updated to kapis/devops.kubesphere.io/v1alpha3.
DevOps is now treated as a regular namespace in the request path, so devops in the path has been changed to namespaces. For example: kapis/devops.kubesphere.io/v1alpha3/workspaces/../devops/../ has been updated to kapis/devops.kubesphere.io/v1alpha3/workspaces/../namespaces/../.
DevOps project member management API has been updated from kapis/iam.kubesphere.io/v1alpha2/devops/../members to kapis/iam.kubesphere.io/v1beta1/namespaces/../namespacemembers.
DevOps project role management API has been updated from kapis/iam.kubesphere.io/v1alpha2/devops/../roles?annotation=kubesphere.io/creator to kapis/iam.kubesphere.io/v1beta1/namespaces/../roles?annotation=kubesphere.io/creator.
DevOps project permission item management API has been updated from kapis/iam.kubesphere.io/v1alpha2/devops/../roles?label=iam.kubesphere.io/role-template=true to kapis/iam.kubesphere.io/v1beta1/roletemplates?labelSelector=iam.kubesphere.io/scope=namespace,devops.kubesphere.io/managed=true.

Misc

Update devops-jenkins authentication method to ks-core OpenId Connect Authentication and remove the dependency on LDAP authentication.

Networking

Gateway v1.0.0

The Gateway is an extension that aggregates services and manages external access to the KubeSphere platform. It has been refactored based on the new microkernel architecture KubeSphere LuBan. It supports one-click enabling and management of cluster gateways, workspace gateways, and project gateways, facilitating comprehensive network configuration at different levels.

Features

Adjust the architecture of the gateway to facilitate decoupling from different vendors' gateways.
Integrate the configuration steps for creating and editing gateways.
Manage all configuration items in the gateway values through editing the gateway YAML.
Configurable display of gateway address when exposed through NodePort.
Support for more granular permission configuration, including gateway management and viewing.

Bug Fixes

Fix the exceptions when exporting gateway logs.

Deprecations

Remove Gateway v1alpha1 CRD.
Remove Nginx v1alpha1 CRD.

API Updates

Add Gateway v2alpha1 CRD.

Misc

Upgrade nginx-ingress from v1.3.1 to v1.4.0.

Network v1.0.0

The Network extension has been refactored based on the new microkernel architecture KubeSphere LuBan. Its functionality is mostly consistent with the network module in KubeSphere Enterprise v3.5.0. It currently includes the management configuration of IPPool and NetworkPolicy, along with some architectural changes and new features.

IPPool no longer uses the previous management approach provided by KubeSphere (ippools.network.kubesphere.io) and directly manages calico ippool (ippools.crd.projectcalico.org) to avoid conflicts with other third-party management tools. It also supports more configurable fields for calico ippool.
NetworkPolicy mainly optimizes the user experience when configuring external whitelists for project network isolation.

Features

Support creating IP pools using YAML and dynamically editing YAML.
Support configuration for more IP pool fields on the UI such as nodeSelector and NatOutgoing.
Support configuring port ranges for external whitelists in project network isolation.
Support configuring multiple network segments and ports for external whitelists in project network isolation.
Support dynamically modifying the configuration and basic information of external whitelists in project network isolation.

Deprecations

Deprecate binding IPPool to workspace, but supports binding native calico ippool to namespaces.
Remove ippools, ipamblocks, and ipamhandles from network.kubesphere.io/v1alpha1.

API Updates

Update ippool to ippools.crd.projectcalico.org/v1. For more API details, please refer to the Network API Reference and Swagger.

Security

Gatekeeper v1.0.0

Gatekeeper is an admission controller for Kubernetes that allows flexible configuration of policies. It uses Open Policy Agent (OPA) to validate requests for creating and updating resources in a Kubernetes cluster.

With Gatekeeper, you can define admission policies flexibly and enforce security admission reviews at the cluster level, ensuring stability and security compliance of your Kubernetes cluster.

Features

Support for configuring security admission policies at the cluster level.

Storage

Storage v1.0.0

The Storage extension includes multiple utility tools related to storage.

This extension deploys and manages the following components:

snapshot-controller: Creates snapshots for PVCs.
snapshotclass-controller: Counts snapshots.
pvc-auto-resizer: Automatically resizes PVCs when capacity is insufficient.
storageclass-accessor: Provides an admission controller to validate whether creating PVCs in a namespace or workspace is allowed.

Enhancements

Improve validation speed of storage class authorization rules.
Faster response for automatic resizing.

Misc

Upgrade snapshot-controller to v4.2.1.

Release Notes