Provides introduction, installation, and management guides for various products, helping users quickly know the basic function and usage of the products.

Documentation

KubeSphere Enterprise is a distributed multi-tenant cloud-native OS developed based on Kubernetes. Powered by feature-rich open-source KubeSphere, KubeSphere Enterprise delivers enhanced features in multi-cloud management, microservice goverance, application management, and other core business scenarios.

KubeSphere Enterprise


- [KubeSphere Core v4.2.0](#kubesphere-core-v4.2.0)
  - [Features](#features)
  - [Enhancements](#enhancements)
  - [Bug Fixes](#bug-fixes)
- [WizTelemetry Observability Platform v2.0.0](#wiztelemetry-observability-platform-v2.0.0)
  - [WizTelemetry Platform Service v2.0.0](#wiztelemetry-platform-service-v2.0.0)
  - [WizTelemetry Monitoring v1.2.0](#wiztelemetry-monitoring-v1.2.0)
  - [WizTelemetry Global Monitoring v1.0.0](#wiztelemetry-global-monitoring-v1.0.0)
  - [WizTelemetry Alerting v1.0.5](#wiztelemetry-alerting-v1.0.5)
  - [WizTelemetry Notification v2.6.1](#wiztelemetry-notification-v2.6.1)
  - [WizTelemetry Network Observability v1.0.3](#wiztelemetry-network-observability-v1.0.3)
  - [WizTelemetry Tracing v1.0.3](#wiztelemetry-tracing-v1.0.3)
  - [Grafana Tempo for WizTelemetry v1.0.0](#grafana-tempo-for-wiztelemetry-v1.0.0)
- [Developer Tools](#developer-tools)
  - [App Store Management v2.1.0](#app-store-management-v2.1.0)
  - [Service Mesh v1.0.2](#service-mesh-v1.0.2)
- [Computing](#computing)
  - [Multi-Cluster Agent Connection v1.1.0](#multi-cluster-agent-connection-v1.1.0)
  - [Metrics Server v0.7.2](#metrics-server-v0.7.2)
- [Edge Computing](#edge-computing)
  - [EdgeWize v3.1.0](#edgewize-v3.1.0)
- [Database and Middleware](#database-and-middleware)
  - [RadonDB DMP v2.2.0](#radondb-dmp-v2.2.0)
- [CI/CD](#cicd)
  - [DevOps v1.2.1](#devops-v1.2.1)
  - [Image Builder v0.1.0](#image-builder-v0.1.0)
- [Network](#network)
  - [Network v1.2.1](#network-v1.2.1)
  - [Gateway v1.1.1](#gateway-v1.1.1)
  - [Ingress Utils v1.0.1](#ingress-utils-v1.0.1)
- [Storage](#storage)
  - [Storage v1.0.1](#storage-v1.0.1)




## KubeSphere Core v4.2.0

KubeSphere Enterprise v4.2.0 is an important milestone in the continuous evolution based on the LuBan architecture. This release features a completely revamped platform UI, simplified resource management hierarchy, and significantly improved usability and operational efficiency.

The version introduces a new design language and user interaction experience. With a more open architecture, it enables deeper integration between the platform and its extensions, greatly enhancing component integration and ecosystem compatibility. The extension mechanism has been further optimized to deliver a consistent and flexible user experience. Core capabilities are more focused, and feature expansion is more flexible, helping enterprises efficiently build a customized cloud-native operating system.

### Features

- Newly designed tenant resource management UI.
- Support for platform maintenance mode.
- Support for user ban policy configuration.
- Support for workload template management.
- Support for user group management.
- Support for extended resource quota configuration.
- Support for API alert prompts.


### Enhancements

- Optimize installation and deployment process of extensions.
- Improve UI for comparing extension configuration changes.
- Support for configuring different image registry addresses for different clusters.
- Optimize Terminal API configuration.
- Improve cascading deletion logic of workspaces.
- Add validation mechanism for RoleTemplate Rules.
- Support for enabling kubeconfig configuration.
- Upgrade dependency libraries and component versions.
- Optimize the management process of member clusters.
- Add 404 and 403 error prompt pages.
- Add `Component Dock`.
- Support for user extended field configuration.
- Optimize ReverseProxy request chain.
- Support mandatory periodic password changes.


### Bug Fixes

- Fix inaccurate application query results.
- Fix potential privilege escalation during application deployment.
- Fix the issue where unpublished applications could not be deployed.
- Fix excessive weight of `global.imageRegistry` configuration during extension deployment.
- Fix potential credential leakage in Cluster API.
- Fix incorrect hook image registry configuration for extensions.
- Fix value validation logic error when setting resource quotas.
- Fix the pagination issue in the Extensions Center.
- Fix several incorrect UI texts.

## WizTelemetry Observability Platform v2.0.0

New extensions: `WizTelemetry Network Observability`、`WizTelemetry Tracing`、`WizTelemetry Global Monitoring`、`Grafana Tempo for WhizTelemetry`.

### WizTelemetry Platform Service v2.0.0

#### Features

- Support eBPF-based network observability, including network topology, service topology (HTTP), and traffic logs (HTTP). (Provided in collaboration with the `WizTelemetry Network Observability` extension).

#### Enhancements

- Update the monitoring query PromQL expression.

#### Bug Fixes

- Fix bug panic when there is no span index.
- Fix bug that logs cannot be sorted when exported.
- Fix the persistent volume usage issue in the workspace overview.
- Fix the problem that the monitoring API cannot limit the query to a specific cluster when querying across clusters.
- Fix the monitoring API unauthorized access.
- Fix the problem that the monitoring API does not support container indicator sorting.

### WizTelemetry Monitoring v1.2.0

In KSE 4.1.x, the Whizard Observability Center will be separated from **WizTelemetry Monitoring** to form a new extension - **WizTelemetry Global Monitoring**, featuring a redesigned and optimized interface. For more details, please refer to **WizTelemetry Global Monitoring**.

#### Features

- Support multi-vendor GPU monitoring including NVIDIA GPU, Huawei Ascend NPU, and Cambricon MLU
- etcd monitoring is enabled by default with automatic generation attempt of monitoring configuration
- The monitoring tab on workload details page displays resource usage and quota comparisons, facilitating adjustments to requests and limits

#### Enhancements

- Improve integration of custom monitoring stacks
- Optimize recording rules configuration in kube-prometheus-stack for enhanced stability

#### Misc

- Update kube-prometheus-stack monitoring stack
- Upgrade Prometheus Operator to v0.81.0
- Upgrade Prometheus to v3.3.1
- Upgrade node-exporter to v1.9.1
- Upgrade kube-state-metrics to v2.15.0 (Note: Due to the update, metrics such as `kube_xx_labels` and `kube_xx_annotations` will be hidden. For more details, please refer to its release note.)

### WizTelemetry Global Monitoring v1.0.0

**WizTelemetry Global Monitoring** is an extension of **WizTelemetry Observability Platform 2.0** that provides **cross-cluster resource monitoring and multi-cluster alerting** functionality. It evolved from the Whizard Observability Center in KSE 4.1.x, undergoing a redesign and interface optimization to become a new standalone extension separated from **WizTelemetry Monitoring**.

#### Features

- Multi-dimensional Resource Drill-down
  
    Provides multiple paths for monitoring resource drill-down, including: 

    Logical Dimensions: Cluster → Project → Workload → Pod, Workspace → Project → Workload → Pod; 
    
    Physical Dimensions: Cluster → Node → Pod

- Multi-dimensional Resource Usage Rankings

  Supports rankings at various levels: Cluster rankings, workspace rankings, project rankings, node rankings, pod rankings;

  Sub-level rankings within the current cluster/ workspace/project/node scope (e.g., project rankings within a cluster, workload rankings within a project)

- Workspace Monitoring with Cross-Cluster Data Aggregation

  Workspace monitoring can aggregate metrics from multiple associated clusters for unified resource usage analysis for the first time.

- Cross-Cluster Pod Search

  Enable fuzzy searches for pods across clusters by name, IP address, project, node, etc., making it easier to locate specific pods.

- Additional Monitoring Views

  New views include:
    - Pod resource usage vs. quota comparison
    - Disk throughput monitoring
    - Network packet loss monitoring
    - Kubelet health monitoring
  
  Interaction improvements, such as direct links from monitoring pages in observability platform to KSE management pages for easier maintenance and updates.

### WizTelemetry Alerting v1.0.5

#### Enhancements

- Add an optional component which depends on the `WizTelemetry Global Monitoring` extension. When the component is installed, `WizTelemetry Alerting` can support global alerting.

### WizTelemetry Notification v2.6.1

#### Features

- The notification history retains all labels and annotations of alerts.

### WizTelemetry Network Observability v1.0.3

The `WizTelemetry Network Observability` extension provides eBPF-based network observability capabilities. It includes the WizTelemetry BPFConductor component, which enables efficient deployment and management of eBPF programs and plugins within Kubernetes clusters. It comes with a set of built-in eBPF programs focused on network observability, providing in-depth visibility into Layer 4 network traffic, Layer 7 HTTP traffic, and network topology.

#### Features

- Support deployment and management of eBPF programs in Kubernetes environments through K8s Custom Resource (CR).
- Support capturing and analyzing Layer 4 network traffic to build global network topology.
- Support capturing and analyzing Layer 7 HTTP traffic to build global service topology and performance monitoring.
- Support parsing of HTTP traffic and exporting traffic logs with detailed request and response information.

### WizTelemetry Tracing v1.0.3

#### Features

- Support deployment of OpenTelemetry Stack as an extension, including OpenTelemetry Operator, OpenTelemetry Collector, OpenTelemetry Demo, etc.
- Support deployment of WizTelemetry Tracing Generator.

### Grafana Tempo for WizTelemetry v1.0.0

#### Features

- Support deployment of Grafana Tempo as an extension.


## Developer Tools

### App Store Management v2.1.0

#### Features

- Restore the display of applications from the default **Stable** repository in the App Store.
- Support one-click synchronization of applications in the system repository to the App Store.

#### Enhancements

- Add support for configuring the **Pod Template** of Jobs during application deployment.
- In multi-cluster deployment mode, custom image repositories can now be configured for installation Jobs.

#### Bug Fixes

- Fix an issue where project admins do not have the permission to create or delete applications.

### Service Mesh v1.0.2

#### Features

- Add support for selecting release strategies when creating or modifying canary releases.

#### Enhancements

- Display virtualservice status for release tasks.

#### Bug Fixes

- Fix the issue where hovering over the "Success" line in traffic monitoring charts does not display the RPS.
- Fix the issue where composed applications remain stuck in the "Updating" status.
- Fix the issue where overly long "Deployment" names in traffic monitoring charts exceed the text box.

## Computing

### Multi-Cluster Agent Connection v1.1.0

- Support configuring tower information directly in the extension configuration.
- Deprecate ConfigMap for Tower configuration and use Secret instead.
- Modularize ks-core's multi-cluster agent connection to support custom agent plugins, decoupling tower from ks-core.

### Metrics Server v0.7.2

#### Features

- UI interaction optimization for creating and managing HPA resources and split into pluggable frontend.
- Add `ks-autoscaling-controller` component to add tags to handle HPA and Workload mapping.

#### Misc

- Upgrade metrics-server from v0.7.0 to v0.7.2

## Edge Computing

### EdgeWize v3.1.0

#### Features

- Support for OTA batch device rapid access.
- Support for cloud-edge collaboration, enabling message communication between edge nodes and cloud clusters.
- Support for image preheating, enhancing application startup efficiency.
 
#### Enhancements

- Support for Luban plugins, expanding ecosystem compatibility.
- Application deployment can expose services on edge nodes through edge network configuration.
- Application deployment supports selecting GPU, NPU, or virtual GPU and virtual NPU resources.
- Optimize automated installation of observability extensions on edge clusters, reducing installation complexity.
- Enhance permission control of observability extensions, improving security and stability.
- Optimize application templates and batch deployment processes, improving deployment efficiency.
- Support for traffic control of cloud-edge communication by instance.
 
#### Misc

- Enhanced support for GPU/NPU resource selection in application deployment.

## Database and Middleware

### RadonDB DMP v2.2.0

- Support custom monitoring/alerting configurations for middleware (via ConfigMap).

## CI/CD

### DevOps v1.2.1

#### Features

- Add support for file management of code repositories.
- Add support for resource manifest management.

#### Enhancements

- Upgrade Jenkins to the latest LTS version.
- Automatically checks repository connectivity when adding repositories via git.
- Reduce the number of podTemplates.

#### Bug Fixes

- Fix incorrect value transmission for "Skip schema validation" during continuous deployment synchronization.
- Fix application deployment failure in member clusters during continuous deployment.
- Fix the issue where users couldn't create kubeconfig-type credentials.
- Fix the pipeline status saving failure.
- Fix incorrect podTemplate image in offline environments.

### Image Builder v0.1.0

#### Features

- Fully restore image builder functionality.

#### Bug Fixes

- Fix potential credential information leakage during source code image building.

## Network

### Network v1.2.1

#### Features

- Support multi-tenant view of NetworkPolicy.
- Support default egress restrictions for project network isolation.
- Support enabling IPPool and NetworkPolicy components separately.

### Gateway v1.1.1

#### Features

- Support replica count limitation for gateways.
- Introduce new API version Gateway v2alpha2.
- Support path rewrite configuration in UI.

#### Enhancements

- Automatic reload configuration after modifying the extension configuration.

#### Bug Fixes

- Fix the issue where project operators could create/modify gateways.
- Fix the inconsistency between the "Gateway Address" displayed on the Ingress list page and the Gateway Settings list page.

### Ingress Utils v1.0.1

#### Enhancements

- Optimize domain reuse validation to allow the same domain under the same gateway.

## Storage

### Storage v1.0.1

#### Features

- Add support for direct binding to NFS underlying volumes when creating PVCs.
- Add support for custom volume initialization.

v4.2.0 Release Notes

KubeSphere Enterprise v4.2.0 Release Notes

KubeSphere Virtualization (KSV) is a lightweight VM management platform built on top of KubeSphere. KSV supports deployment in single-node and multi-node modes and provides easy-to-use services to meet enterprise-grade requirements of virtualization.

KubeSphere Virtualization


KubeSphere Virtualization (KSV) is a lightweight VM management platform built on top of KubeSphere. KSV supports deployment in single-node and multi-node modes and provides an enterprise-grade service of virtualization to meet business needs of individual and enterprise users in diverse scenarios.  KSV can help you reduce hardware costs in early stages and allows you to create VMs based on business requirements. This way, you can develop and deploy applications in an efficient manner.

On top of the cloud-native architecture Kubernetes, KSV is more lightweight, flexible, and open than legacy virtualization services. KSV can also provide capabilities of service self-healing, auto scaling, and flexible scheduling.

## Benefits<a href="" id="e054f4aee3ad4d92baee4f2f0a9eb9e5" class="bare"></a>

KSV can provide compute and storage resources with the following benefits:

-   **Low cost**: supports a minimum specification of 4 CPU cores and 8 GB memory and multi-node clusters.

-   **Simple operation**: provides an easy-to-use web console to help you monitor all resources for enterprise hardware virtualization and enables service self-healing.

-   **Quick installation**: allows you to deploy and update a cluster in 20 minutes.

-   **Quick response**: allows you to create and resize VMs within seconds.

-   **Excellent compatibility**: frees you from the dependency of the underlying infrastructure. You can create VMs on any servers that suppport virtualization.

-   **Connection to containers**: seamlessly connects to KubeSphere and integrates VMs and containers.

## Architecture<a href="" id="ee18416cd47e4ca3bb1baec5805d5675" class="bare"></a>

On top of the cloud-native architecture of Kubernetes, KSV abstracts away the complexities of the infrastructure and basic configuration. You can run KSV in development or testing environments of various types, from private clouds, public clouds, VMs, physical servers, to edge nodes. KSV provides a range of functional modules, such as physical nodes, networks, storage, VMs, images, disks, and security groups.

 The following figure shows the architecture.

<img src="/images/docs/ksv/introduction/architecture-en@0a2929d0392f4ae7a86bea4db763940a@.png" alt="35px" />

## Underlying logic<a href="" id="62a2ab6421074eea9a35382520c8bc1f" class="bare"></a>

The following table describes the underlying logic of KSV.

<table class="tableblock frame-all grid-all stretch">
<colgroup>
<col style={{width: "20%"}} />
<col style={{width: "80%"}} />
</colgroup>
<thead>
<tr class="header">
<th class="tableblock halign-left valign-top">Module</th>
<th class="tableblock halign-left valign-top">Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
VMs
</div>
</div></td>
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
By using the KubeVirt plug-in, KSV allows you to manage the lifecycle, status, and connection of each VM.
</div>
</div></td>
</tr>
<tr class="even">
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
Networks
</div>
</div></td>
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
KSV runs on top of containers, which requires more NICs for networking. KSV allows you to use NICs on VMs in containers. At the underlying layer, KSV uses Multus to enable multiple NICs for containers. KSV supports Macvtap for VMs and integrates Kube-OVN to enable software-defined networking (SDN). Kube-OVN provides underlay and overlay network types. You can select a network type based on business requirements.
</div>
</div></td>
</tr>
<tr class="odd">
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
Images
</div>
</div></td>
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
KSV uses Contexts and Dependency Injection (CDI) to import images into persistent volumes (PVs) and creates system disks by cloning PVs.
</div>
</div></td>
</tr>
<tr class="even">
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
Storage
</div>
</div></td>
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
KSV can integrate into the Kubernetes Container Storage Interface (CSI) for storage resource management. KSV uses open-source Ceph block devices and can connect to commercial storage systems by using CSI.
</div>
</div></td>
</tr>
<tr class="odd">
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
Security groups
</div>
</div></td>
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
KSV integrates Kube-OVN and enables security groups by using the NICs powered by Kube-OVN.
</div>
</div></td>
</tr>
<tr class="even">
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
Bandwidth limits
</div>
</div></td>
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
KSV integrates Kube-OVN and enables bandwidth limits by using the NICs powered by Kube-OVN.
</div>
</div></td>
</tr>
<tr class="odd">
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
Monitoring metrics
</div>
</div></td>
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
On top of the native Kubernetes Prometheus monitoring system, KSV allows you to configure, collect, store, and query cluster monitoring data.
</div>
</div></td>
</tr>
<tr class="even">
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
Alerts and monitoring
</div>
</div></td>
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
On top of the native Kubernetes Prometheus monitoring system, KSV allows you to configure, collect, store, and query cluster alert data.
</div>
</div></td>
</tr>
<tr class="odd">
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
Log queries
</div>
</div></td>
<td class="tableblock halign-left valign-top"><div >
<div class="paragraph">
On top of the native Kubernetes Fluent Bit and Elasticsearch, KSV allows you to collect, store, and query log data.
</div>
</div></td>
</tr>
</tbody>
</table>

## Billing<a href="" id="618711fb374d4b6a9cea0a57f96f8935" class="bare"></a>

KSV provides the following types of editions: Basic, Developer, and Enterprise. The Basic Edition is suitable for individual and micro enterprises. In this edition, you can deploy up to 3 nodes free of charge. The Developer and Enterprise editions provide more comprehensive virtualization services and allow middle and large enterprises to deploy more nodes. For more information about billing and capabilities of each edition, see [Editions](../../01-introduction/03-editions/#7380f54441404ccabd94b37339318894).

## Pre-deployment checklists<a href="" id="15b938d2fdec4a78a9626d9ecc6293fd" class="bare"></a>

Before you deploy a KSV cluster, use the following checklists to ensure you have a prepared environment:

-   Server nodes

    KSV supports Linux and the kernel version must be 4.0 or later. We recommend that you use one of the following OSs: Ubuntu 18.04, Ubuntu 20.04, CentOS 7.9, Unity Operating System (UOS), Kylin V10, and EulerOS.

-   Network planning

    We recommend that you plan two networks for the following use:

    -   Management network: the network where the server nodes of KSV resides. We recommend that you use private VLANs and connect switches using trunks.

    -   Business network: the network where virtual resources reside. You can configure multiple VLANs based on business requirements and connect vSwitches using trunks.

-   Security

    KSV provides security groups to control inbound and outbound traffic of VMs in the security groups. This improves the security of virtualization services. For more information, see [Security groups](../../03-user-manual/08-security-groups/#700ba3ce4e964f8cba07684e2cf1b773).

    To ensure business continuity, KSV also provides snapshots for backup. We recommend that you create snapshots for disks and VMs to enable disaster recovery.


Describes the features and scenarios of KSV.

What is KSV?

KubeSphere Backup is a SaaS service that helps you back up your Kubernetes cluster resources to your object storage repositories.

KubeSphere Backup


KubeSphere Backup is a Disaster-Recovery-as-a-Service service developed by the KubeSphere team for use cases related to hybrid clouds. You can back up and recover the container resources in your Kubernetes clusters through several clicks on the console with no need to deploy the infrastructure for backup and recovery on your own.

## Features

KubeSphere Backup provides easy-to-use wizard UI to simplify backup and recovery operations, facilitates the establishment of backup plans and recovery plans, and improves the overall disaster tolerance. 

## Use cases

KubeSphere Backup is applicable to the following use cases:

- **System or hardware failure**

   Reduces the risks of business interruption due to system or hardware failure to ensure the continuity of core businesses running in your Kubernetes clusters.  

- **Development and testing environment changes**

   Recovers the data backed up in any storage repository at any time in case of development and testing environment changes or configuration modifications.

- **Malicious attacks or improper operations**

   Prevents data loss due to malicious attacks or improper operations by backing up and recovering data to ensure data integrity.

- **Protection across multiple clusters and storage repositories**

   Implements data backup and recovery across multiple clusters running by different cloud providers, in different locations, and on different infrastructures according to your needs.

- **Container data migration across clouds**

   Provides an easy-to-use method for data migration to help you avoid vendor lock-in.





Introduces the features and use cases of KubeSphere Backup.

What is KubeSphere Backup

KubeSphere Lite is a SaaS solution on KubeSphere Cloud, where you can create a managed Kubernetes cluster and quickly get started with KubeSphere.

KubeSphere Lite


KubeSphere Lite is a SaaS solution launched by the KubeSphere team. On the KubeSphere Cloud platform, users can create a managed Kubernetes cluster with a few clicks and quickly get started with feature-rich KubeSphere.

Describe features and application scenarios of the KubeSphere Lite on KubeSphere Cloud.

Overview

Built on top of open-source cluster inspection tool KubeEye, KubeSphere Inspector is a cluster inspection SaaS service designed for multi-cloud Kubernetes environments.

KubeSphere Inspector


Built on top [KubeEye](https://github.com/kubesphere/kubekey), an open-source automatic inspection tool, KubeSphere Inspector is a cluster inspection SaaS service designed for multi-cloud Kubernetes environments. It helps users scan cluster vulnerabilities to detect possible risks and enhance cluster security, which ensures that Kubernetes clusters run securely, stably, and efficiently.

KubeSphere Inspector is applicable to the following use cases:

- **Ensure Kubernetes security**

   Eliminates Kubernetes vulnerabilities and blind spots and leverages DevSecOps to ensure Kubernetes and container security.

- **Meet compliance and security audit requirements**

   Detects whether security configurations and permission management comply with regulations to prevent potential problems from affecting security audit.

- **Diagnose cluster abnormalities**

   Detects cluster abnormalities in time and makes optimization suggestions to ensure that clusters always run stably and efficiently.

- **Reduce costs**

   Collects resource consumption statistics and makes resource optimization suggestions to prevent resource overuse and reduce the cost of using Kubernetes.




Describes features and use cases of KubeSphere Inspector.