CHANGELOG.md

Release v3.3.1 - 2023/12/25

UPGRAGDE NOTICE

UPGRAGDE NOTICE If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. if you CubeFS version is v3.2.1 or before, and need to upgrade to v3.3.*, you must follow these upgrade steps:

1. When upgrading the metanode node, you need to add "raftSyncSnapFormatVersion": 0 to the configuration file.
2. After all metanodes are upgraded, remove this configuration item so that raftSyncSnapFormatVersion defaults to 1.
3. Restart all metanodes.
4. Please upgrade the client at the end, the master has enhanced the check of the client, otherwise it will cause the client to mount abnormally

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• client: Add a trash feature to client (#2291, @bboyCH4)

Enhance

• datanode: Support cleaning up garbage data on the datanode (#2880, @Victor1319)
• metanode: Audit log in metanode (#2899, @NaturalSelect)
• datanode: limit datanode's disk flow and concurrent iops (#2900, @sejust)
• client: Too many waring errors log when use trash (#2915, @bboyCH4)

Bugfix

• client: Audit's writer maybe nil leadto error (#2901, @longerfly)
• gapi: fix read body entirely into memory (#2691, @tangdeyi)
• objectnode: Timing attack can leak user passwords and CubeFS leaks users key in logs from cncf security audit (#2781, @leonrayang)
• client: Optimize insecure random number generation in function util/string.go:RandomString (#2698, @true1064)
• docs: Add Security Best practice (#2853, @leonrayang)
• gapi: Timing attack can leak user passwords and CubeFS leaks users key in logs from cncf security audit (#2781, @leonrayang)
• objectnode: fix limiter lib deadlock (#2794, @tangdeyi)
• master\objectnode: some commits related with security audit (#2824, @leonrayang)
• master: qos.Lock of assignClientsNewQos forget release and trigger deadlock(#2861, @leonrayang)
• bcache: fix possible deadlock of bacahe(#2819, @longerfly)
• ClusterMgr: fix volume manager deadlock(#2793, @tangdeyi)
• blobnode: fix put shard deadlock (#2790, @mawei029)
• client: inode leak when use trash(#2911, @bboyCH4)
• client: trash encounters IO error when dealing with log file names(#2912, @bboyCH4)
• client: trash delete interval do not work(#2914, @bboyCH4)
• client: Failed to delete a large directory with trash enable(#2917, @bboyCH4)

Release v3.2.1 - 2023/03/16

UPGRAGDE NOTICE

UPGRAGDE NOTICE If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• meta\client: add quota of children dentries for directory (#1763, @wuchunhuan )
• master: add qps limiter for individual API on master (#1766, @wuchunhuan )
• client: add audit log for fuse client (#1764, @wuchunhuan )
• objectnode\sdk: objectnode supports docking erasure-code subsytem(blobstore) (#1765, @wuchunhuan)
• master\datanode: Add speed control of dp decommission for disk or datanode (#1778, @bboyCH4)

Enhance

• master: master support leader switching by external trigger (#1772, @leonrayang )
• metanode\master: metanode support follower read (#1775, @leonrayang)
• master: Add metrics for meta-partition consistency detection (#1776, @leonrayang)
• master: add API statistics log for master (#1767, @wuchunhuan)
• datanode: Datanode startup should not be influnced by missing data partitions (#1773, @leonrayang)
• master: volume's capacity must be bigger than 120% used size (#1723, @Victor1319)
• metanode: metanode support config memRatio of the machine. (#1725, @Victor1319)
• datanode: datanode support config diskPathPrefix to manage disk path list. (#1727, @Victor1319)
• metanode: reduce extent alloc memory when metanode snapshot (#1728, @Victor1319)
• metanode: random write may cause metanode memory grow (#1729, @Victor1319)
• datanode: bad disk space is calculated in datanode's total space (#1601, @Victor1319)
• datanode: no permission disk can't be detected (#1602, @Victor1319)
• master\cli: add display of unavailable replica for cfs-cli datapartition check (#1771 ,@true1064)
• datanode: datanode should compute datapartition used size right after loading extents from disk (#1782 ,@true1064)
• client:enhance support nfs access operation (#1798, @leonrayang )
• master: add some metircs (#1785, @liubingxing @litao)
• master\datanode\client\cli: fix the typos and format code (#1787, @liubingxing @litao)
• datanode: add lack datapartitions monitor (#1789, @guojunhao)
• master: create data partition exclude decommissioned disk (#1790 , @guojunhao)
• datanode: Optimize the qos for random write (#1791, @litao)
• metanode: speed up metanode startup (#1792, @liubingxing)
• cli: add more info to cli/fsck cmd (#1793, @liubingxing @litao)

Bugfix

• master: master snapshot recover not reset local rocksdb info (#1522, @wuchunhuan )
• master:Memory cost too fast during restart in case of data partition‘s count is magnity (#1774 , @leonrayang)
• client: Readonly dp can still accept write request from client (#1779, @bboyCH4)
• metanode: Metanode should not establish connection to blobstore for cold volume (#1781, @bboyCH4)
• client: blockcache service may be oom when the client caches many large files concurrently (#1783, @zhangtianjiong)
• datanode: too may delete requests may cause a lot of tcp connections. (#1724, @Victor1319)
• master: when master creating data partition, should use vol.dataPartitionSize as datapartition.total (#1777 ,@true1064)
• master: In addition to all replica status is readwrtie, still need to consider volume used space before check set datapartition status as readwrite (#1780 ,@true1064)
• master: master auto creates dataparitions when volume goes from full to not full (#1784 ,@true1064)
• unit test: Fix the pipeline cubefs-ci error problem (#1795, @baijiaruo)
• metanode\client:do rename on the file which is soft link not worked well (#1797 , @leonrayang)
• meta\master: fix some bug to improve cluster stability (#1786, @liubingxing @litao)

Release v3.2.0 - 2022/10/14

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.

Main Feature

• Merge data subsystem blobstore(ec) to main branch.
• blobstore: Merge some service modules #1563

Enhance

• blobstore: ClusterMgr support simple key-value persistent storage #1566
• blobstore: Remove blobstore's dependency on monogdb #1493
• blobstore: Remove blobstore's dependency on consul #1507
• blobstore: Support audit log filtering #1506

Bugfix

• blobstore: Blobnode use the same erasure encoder and buff-pool libraries as Access #1498
• blobstore: Separate flow control for repair and migration traffic #1508
• blobstore: Retry http body under rpc request #1567

Release v3.1.2 - 2022/10/13

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Bugfix

• master: add leaderSize param in create partition req #1456

Enhance

• master: support decommission broken replica for 2-replica no leader dp #1456
• datanode: in case disk full, repair size is also thought as used size #1456
• raft: raftstore.monitor can be optimized #1543

Release v3.1.1 - 2022/10/08

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Bugfix

• metanode: Show metapartition id correctly #1554
• metanode: remove redundant memcopy when reading raft snapshot #1552

Enhance

• master: update metanode or datanode id need drop old one before #1555
• metanode: add metric mpDentryCount #1553
• datanode: fallocate return interrupt err try again #1551

Release v3.1.0 - 2022/08/23

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Main Feature

• master\client\datanode: ：Provide QoS services to enhance multi-tenant isolation #1447
• client: Caching acceleration #1446
• master\datanode: Support two replicas data storage #1389
• master: Add vol label in cfs_metanode_mpInodeCount #1482
• client: Support config posixAcl for volmue #1485
• master: Add datapartition count limit for datanode. #1481

Bugfix

• datanode: TinyDeleteRecord file will be very big when one replica is down #1433
• client: EnablePosixACL is not vaild in centos system #1393
• blockcache:Blockcache memory overflow and be killed #1422
• yum source: Yum install script fails due to missing packages #1410
• unit test: S3test random path busy to produce data and timeout #1515
• master: Zone name check not strict enough while crossZone enabled #1479
• datanode\master: Avoid writing operation failed because of disk full #1519

Enhance

• master: Enable volume update from 3 to 2 replics #1516

Release v2.5.2 - 2022/06/27

UPGRAGDE NOTICE If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Main Feature

• master: support to delete data replica with force #1258

Bugfix

• master: revert the deletion process in the tiny extent to avoid deletion failure caused by different logical and physical sizes #1439
• master: add metapartition replica check to fix replicas status still writeable while metanode unavalible #1440

Release v3.0.0 - 2022/04/22

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Notice

This is a significant release based on master branch with a lot of code mainly to support multi layer data layer including blobstore (Erasure-Code) storage, blobstore supports erasure coding storage method(https://github.com/cubeFS/cubefs-blobstore) to support low frequency volume and reduce cost .

Before this we released v3.0.0-beta based on release v2.4.0 (https://github.com/cubeFS/cubefs/releases/tag/v3.0.0-beta) , which make volume can be create as standard(3 replica) or low frequency access type(Erasure-Code), but not support multi data layer.

Release formal version release-v3.0.0 which based on the newest master commit, below is a list of the main features.

1. BlobStore subsystem which supports erasure coding storage. #212
2. multicache layer, 1st level:client local read cache; 2nd level:hot pool replica; 3nd level:code pool based on erasure code. #1291
3. single data replica supported. releated project: https://github.com/oppo-bigdata/shuttle used for remote shuffle. #1227
4. client hot online restart(experimental).
5. optimization reduce extents fragmentation. #1318
6. manage third part library with go mod.

Please refer to the documentation for details,the content already updated.

• English version: https://cubefs.io/docs/master/overview/introduction.html
• Chinese version: https://cubefs.io/zh/docs/master/overview/introduction.html

Main Feature

• log: support write ebs log to file and update ebs log level #212
• client\datanode\master: support one replica of data storage #1227
• master: support get master's raft status info #1227
• master: support preload data from blobstore to cubefs #212
• client: client local read cache #212
• client: Time-consuming statistics of client sdk interface
• client: client support blobstore #212
• client: support ebs index #212
• master: support different type dataparitition(normal dp,cache dp,preload dp) #212
• master: support mulit volume type(replcias volume,ec volume) #212
• project: enable go mod and update vendor #1343
• datanode: intruduce metrics degrade level to datanode #1438
• master: not do ttl & updateSize for hot vol #212
• compile: change go mod version from 1.14 to 1.16 and consistent with compile #1343
• compile: enable go mod and update vendor #1343
• project: change from chubaofs to cubefs #1343

Bugfix

• master: single replica create volume should set followeread with true #1227
• raft: raft snapshot still running while peer remove happened which lead to the snapshot gorutine cann't end. #1227
• master: Refactor finding partitins to be offlined #1189
• master: Avoid race syncing node's info when setting rdonly #1189
• client: follow recover handlers when closing open handler #1316
• master: remove 5 GB left limit for disk to create datapartition #1189
• server: if server recive kill-signal before being running-state, server will can't be killed graceful #1261

Enhance

• master: not do ttl & updateSize for hot vol #212
• master: support disk, datanode, metanode decommission
• client: add cli-command for cold volume #212
• master: ec-volume can't be deleted if size not equal 0 #212
• client: add cfs-cli request parameters #212
• client: add scan cli #212
• master: add volume type to newVolCreateCmd #212
• master: not support force delete cold volume #212
• datanode: add cfg diskRdonlySpace for datanode #1353
• datanode: use os.ReadDir instead of ioutil.ReadDir #1438
• datanode: improve performance of write. data allocated didn't put back to buffer pool #1438
• datanode: reduce datanode cpu usage #1438
• datanode: remove unnecessary json unmarshal when getting local extents #1438
• datanode: reduce datanode cpu usage by eliminate file seeks #1438
• client: mitigate the pain of extents fragmentation #1318

Release v2.5.1 - 2022/01/26

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Notice

• this release mainly used to optimize datanode cpu usage

Feature

• datanode feat: intruduce metrics degrade level to datanode [#1234]

Bugfix

• datanode remove 5 GB left limit for disk to create datapartition[#1345]
• fuse client follow recover handlers when closing open handler[#1344]

Enhance

• datanode improve performance of write. data allocated didn't put back to buffer pool [#1346]
• datanode remove unnecessary json unmarshal when getting local extents [#1346]

Release v2.5.0 - 2022/01/06

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Main Feature

• fuse client:support auto push data to push gateway
• master:reduce flow from interface of "client/partiton" that follower support it and will not redirect to leader
• raft:the receive buffer channel size of raft support to be configurable
• metanode:implement the content summary
• master:domain for cross zone

Bugfix

• fuse client:fix: currAddr of stream conn is empty
• fuse client:fix: update extent cache no matter whether eh is dirty or not
• metanode:when kill metanode(mean while mp stoped firstly) and if apply snapshot happen at the same time, snapshot will be block, causeing metanode can't be killed
• metanode:meta node migration be recorded in badmetapartitions concurrently without lock which lead to mp id miss
• metanode:return file nodes for statfs
• metanode:meta not use warn when consulMeta not set
• metanode:makes AppendExtentKeyWithCheck request idempotent
• fuse client:push addr shadowed if export port is not set
• objectnode:readdir gets insufficient dentries if prefix larger than marker
• raft:remove redundant memcopy when reading raft snapshot
• raftstart tcp listen before starting raft
• ltp test:broken test case ftest01，update and unlock ltp test cases
• ltp test:fix ltptest ci bug
• docker:finstall killall command in the docker image
• cli:fix cli tool typo

Enhance

• master: support disk, datanode, metanode decommission and assign target node
• master:return response cache directly, not copy again
• master:add vol usedRatio warn log when usage > 90%
• master:add master metrics vol_meta_count to export vol dp/mp/inode/dentry
• master:del replica with force if decommission hang
• metanode:add log for loading meta partitions
• fuse client:increase client retry times to avoid mp raft election timeout.
• grafna: change grafana volume used size rate from rate to deriv
• grafna:support push monitor data to gateway
• objectnode: add subdir authorized check for objectnode
• objectnode: tracking objectnode modification for cfs-server
• raft:check nil when get leader term info
• raft: check size when read data from heartbeat port
• raft: add getRaftStatus for metanode
• cli:update gitlab-ci
• cli:Enhancement: add ci check for ltptest result
• cli:enhance: upload docker_data when ci tests finish
• cli:Create and Update ci.yml
• style:rename cfs to cbfs & optimize the libsdk module
• docker:change: use ghcr instead of dockerhub
• build: add version information and rules for building fsck
• build:change: update build status badge

New feature brief introduction

Fault domain(Failure domain)

1. Purpose In the cross zone scenario, the reliability need to be improved. Compared with the 2.5 version before, the number of copysets in probability can be reduced. The key point is to use fault domains to group nodesets between multiple zones.

• Reliable papers https://www.usenix.org/conference/atc13/technical-sessions/presentation/cidon

• Chinese can refer to https://zhuanlan.zhihu.com/p/28417779

2. configuration

1) Master

• Config file：master.json Enable faultDomain set item "faultDomain": true

• Zone count to build domain faultDomainGrpBatchCnt，default count:3，can also set 2 or 1

  If zone is unavaliable caused by network partition interruption，create nodeset group according to usable zone Set “faultDomainBuildAsPossible” true, default is false

  The distribution of nodesets under the number of different faultDomainGrpBatchCnt 3 zone（1 nodeset per zone） 2 zone（2 nodesets zone，1 nodeset zone，Take the size of the space as the weight, and build 2 nodeset with the larger space remaining） 1 zone（3 nodeset in 1 zone）

• Ratio 1) The use space threshold of the non-fault domain（origin zone） After the upgrade, the zone used by the previous volume can be expanded, or operated and maintained in the previous way, or the space of the fault domain can be used, but the original space usage ratio needs to reach a threshold, that is, the current configuration item The proportion of the overall space used by meta or data Default：0.90 UpdateInterface： AdminUpdateZoneExcludeRatio = "/admin/updateZoneExcludeRatio"

  2) the use space threshold of the nodeset group in the domain Nodeset group will not be used in dp or mp allocation default：0.75 Update interface： AdminUpdateDataUseRatio = "/admin/updateDomainDataRatio"

2) Datanode && metanode

After the fault domain is enabled, a minimum configuration of the fault domain is constructed under the default configuration: Each zone contains 1 datanode and 1 metanode, and the zone name needs to be specified in the configuration file There are 3 datanodes and 3 metanodes in 3 zones

For example, three datanodes (metanode) are configured separately: "zoneName": "z1", "zoneName": "z2", "zoneName": "z3", Start after configuration, the master will build a nodeset for z1, z2, and z3, and component a nodesetgrp

3. Note 1) After the fault domain is enabled, all devices in the new zone will join the fault domain 2) The created volume will preferentially select the resources of the original zone 3) Need add configuration items to use domain resources when creating a new volume according to the table below. By default, the original zone resources are used first if it’s avaliable

| Cluster:faultDomain | Vol:crossZone | Vol:defaultPriority | Rules for volume to use domain | h| ------ | ------ | ------ |------ | | N | N/A | N/A | Do not support domain | | Y | N | N/A | Write origin resources first before fault domain until origin reach threshold | | Y | Y | N | Write fault domain only | | Y | Y | Y | Write origin resources first before fault domain until origin reach threshold |

Note: the fault domain is designed for cross zone by default. The fault domain of a single zone is considered as a special case of cross zone, and the options are consistent

example : curl "http://10.177.200.119:17010/admin/createVol?name=vol_cross5&capacity=1000&owner=cfs&crossZone=true&defaultPriority=true"|jq .

Content Summary

1. Purpose In order to query the content summary information of a directory efficiently, e.g. total file size, total files and total directories, v2.5 stores such information as the parent directory’s xattr.

The parent directory stores the files, directories and total file size of the current directory. Then only need to make recursive of the sub directories, and accumulate the information stored by the directories to query the content summary information of a directory.

2. Configuration Client config file: fuse.json 1) Enable XAttr ”enableXattr”:”true” 2) Enable Summary ”enableSummary”:”true” Both of xattr and summay have to be set if you want to mount a volume to the local disk. Set summary is enough if you want to access the volume via libsdk.so.

3. How to use There are two different ways to get the content summary of a directory. 1) Fuse mount getfattr -n DirStat yourDirPath getfattr can be installed by: yum install attr or apt install attr 2) libsdk.so cfs_getsummary (libsdk/libsdk.go)

4. Note 1)The incremental files’ summary information will be held by their parent directories. But the old files will not. Use cfs_refreshsummary (libsdk/libsdk.go) interface to rebuild the content summary information. 2)The files, directories and total file size are updated asynchronously in the background. Users are not aware of these operations, but it does increase the requests to meta servers (usually doubled). You are recommended to evaluate the impact to your cluster before using this feature.

Release v2.4.1 - 2021/12/31

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.

Feature

• meta&object introduce ReadDirLimit interface to retrieve partial results #1234
• fuse client use ReadDirLimit in fuse client #1244

Bugfix

• sdk makes AppendExtentKeyWithCheck request idempotent #1224
• meta start tcp listen before starting raft 1256
• raft remove redundant memcopy when reading raft snapshot to avoid snapshot hanging 1264
• object handling range read request in a behavior compatible with S3 #1286 #1298

Enhance

• sdk add version in the http requests issued to master to collect client info 1262
• sdk mitigate the pain of extents fragmentation 1282

Release v2.4.0 - 2021/05/14

UPGRAGDE NOTICE

If your CubeFS version is v2.3.x or before, and need to upgrade to v2.4.x , please follow the following upgrade steps:

1. first of all, firewall open two more port (17710 & 17810) in your machine to support tcp multiplexing；

   17710 is calculated by 17210, the port listened by metanode, add 500
   17810 is calculated by 17310, the port listened by datanode, add 500
   

2. then add item '"enableSmuxConnPool"=false' in your datanode's config file, upgrade all datanodes; detail steps as follow:

   a. using new dataNode bin file replace the old version
   b. add  '"enableSmuxConnPool"=false' in your datanode's config file
   c. restart datanode process
   d. update all the datanodes in the custer
   

3. update '"enableSmuxConnPool"=true' in your datanode's config file, restart all datanode again;

   notes: before set enableSmuxConnPool as true, you must finsh step 2. 
   

4. upgrade all metanode bin file and restart meta process;

5. upgrade all master bin file and restart master process;

6. upgrade all client bin file and restart client process;

The key point of upgrading from previous versions to v2.4.0 is to make sure ALL datanodes are upgraded to the new version with "enableSmuxConnPool" set to "false" before upgrading other components, such as metanode, master and client. Then datanode can be configured to enable smux conn pool, i.e. "enableSmuxConnPool" set to "true".

Feature

• dataNode smux support #1124
• metaNode smux support #1124
• datanode metanode support TickInterval configurable #1117

Refector

• master remove token-based volume read-write permission control #1119

Bugfix

• datanode fix build repair task panic #1124
• metanode too many open/close stream when delete limit #1124
• master do split on metanode partion if inode count larger than maxinodecnt #1124
• datanode dp.raftPartition nil pointer #1124
• datanode fix datanode repair log print error #1124
• master partition be split can be write if not full and less then on step #1124
• metanode new a gauge instance every time to avoid a gauge instance modified concurrently #1124
• master fix conflict for histogram data when report data concurrently #1124
• datanode fix bug when packet opCode=OpExtentRepairRead and network is unstable #1124

Enhance

• client export init should be after init log in fuse client #1124
• util flush log before os.exit #1124
• datanode format datanode log when delete #1124
• metanode update log level to info if mp is not exist #1124

Release v2.3.0 - 2021/02/26

UPGRAGDE NOTICE

Note that when upgrading from versions prior to v2.3.0 to version v2.3.0 or beyond, servers(i.e. resource manager, metanode and datanode) must be upgraded before client(i.e. fuse client and objectnode) due to #1098.

Feature

• client: support multiple subdir permissions for an individual user. #1051
• sdk: introducing libsdk so applications can embed the use of cubefs to a single binary and no additional process is required at runtime. #1082

Enhancement

• datanode: improve the shut down process of datanode so the raft apply id is persisted. #1030
• raft: auto fix crc-mismatch raft log crc. #1039
• improve stability of the whole system. #1098

Bug fix

• objectnode: fix key encoding issue of S3 ListObjects API. #953
• datanode: fix data partition decommission timeout. #972
• metanode: change mtime of the directory when creating or deleting dentry. #1000
• datanode: fix statfs deviation when using EXT4 as the underlying local filesystem for datanode. #1031
• client: func sortHostsByDistance change dp's hosts. #1106

Release v2.2.2 - 2020/09/22

Feature

• datanode: Introducing data partition selector which allow users to customize the client's selection logic for data partition when reading and writing, and can switch at any time. #853

Enhancement

• client: Add enablePosixACL configuration to enable POSIX ACL feature. #906
• datanode: Improved data recovery speed. #899

Bug fix

• datanode: Fix the issue that the information in the memory is not released immediately after deleting the extent, which causes the status information of the data partition to be incorrect. #938
• datanode: Fix the issue that repeatedly sending requests to the failed data partitions. #937
• objectnode: Fix the issue that recursive making directory concurrent with sample multi-layer path prefix. #904

Release v2.2.1 - 2020/09/07

Bug fix

• master: Fix the concurrency safe issue when removing raft member. #893
• sdk: Fix the panic issue while remove data partition concurrently. #894

Release v2.2.0 - 2020/09/01

Enhancement

• object: Implemented S3 api 'DeleteBucketPolicy' for bucket deletion. #757
• master: Introducing management API for volume capacity expanding and shrinking. New capacity must be set more than 20% lager than used. #764
• master: Introducing management API for updating node address. #813
• master: Introducing management API for checking nodes. #813
• metanode: Add header and checksum verification for EXTENT_DEL files. #813
• cli: Add logging for CLI tool, and added several commands to call master API. #764 #801
• compile:Support direct compile or docker cross compile on Arm64 platform. #779
• client: Introducing nearRead option that allow client read data priority from the nearest DataNode to improve reading performance. #810
• client: Update volume follower read config from master periodicity. #837

Refactor

• object: Improved compatibility for ListObjects and ListObjectsV2 interfaces. #769
• master: Check whether the used space between the replicas is consistent when performing the load partition operation. #813
• metanode datanode: Optimize batch delete Extent; add autoRepairLimitRater on DataNode. #781
• datanode: Introducing autoRepair option for limit data repair speed on DataNode. #842
• datanode: The data repair task will skip process extent file which has been already deleted. #842

Bug fix

• master: When loading metadata of cluster, using the current ID instead of the loaded ID. #821
• metanode: Fix the deadlock problem while MetaNode deleting dentry. #785
• metanode: Fixes #760 Add inode to freeList when NLink is 0, and delete inode 7 days later. #767
• datanode metanode: When DataNode and MetaNode start, the partitions not exist in cluster view will be renamed to expiredPartition and skip loading. #824
• datanode metanode: Fixes issue#698, Raft instance delete itself by applying ConfChange raft log. #866
• metanode: MetaNode may not free space. #838
• client: Fix the problem that client opening file with outdated extents information. #783
• client: Batch inode get mechanism is out of service. #847
• client: Fix the problem that makes authorized user mount volume failure. #828
• datanode: Fix several issue in automatic repair process for tiny extent. #855 #857
• datanode: Fix the problem that the client still applies to create a new extent to a data partition that has no space. #867

Document

• Updated Q&A, environment&capacity planing documentation. #801

Release v2.1.0 - 2020/07/09

Feature

• console: CubeFS Console is a web application, which provides management for volume, file, s3, user, node, monitoring, and alarm. The CubeFS Console makes it much easier to access services provided by CubeFS. #728
  Please refer to https://cubefs.readthedocs.io/en/latest/user-guide/console.html to start the Console.
• metanode: Provide compatibility verification tool for meta partition. #684
• object: CORS access control. #507
• fuse: Introduce fsyncOnClose mount option. Choose if closing system call shall trigger fsync. #494

Optimize Memory Usage

Release2.1.0 did a lot of work to optimize memory usage.

• Modify the max write size to 128k; limit the rate of Forget requests. #533
• If concurrent write operations are more than 256, bufferPool is no longer used. #538
• Eliminates the unnecessary memory allocations for Inode struct by using proto InodeInfo struct directly. #545
• Use sync Pool and rate limiter to limit the memory usage. #639
• Use total count instead of rate. #642
• Uses sorted extents instead of BTree. #646

Enhancement

• master: Add set/get deleteBatchCount interfaces. #608
• master: Delete partitions on DataNode/MetaNode concurrently in decommission action. #724
• master: Added api for getting & setting parameters for batch deleting extents. #726
• metanode: Filter inode candidates that will be sent to a partition in batch-inode-get to save memory. #481
• metanode: Batch delete inode, unlink&evict dentry. #586
• datanode: Prioritize healthy data node when sending message to data partition. #562
• metanode datanode: Check expired partition before loading. #624
• object: Implemented several conditional parameters in GetObject action. #471
• object: Making S3 credential compatible with earlier version of CubeFS. #508
• object: Totally support presigned url; Partial support user-defined metadata; Making ETag versioned. #540
• object: CopyObject across volumes; Support coping directory; Support coping xattr. #563
• object: Parallel downloading. #548
• object: Add 'Expires' and 'Cache-Control' in putObject/copyObject/getObject/headObject. #589
• object: Modify part system metadata. #636
• fuse: Decrease request channel size to save memory. #512
• fuse: Let the client daemon inherit all the environment variables but not just PATH. #529
• fuse: Introduces a debug interface /debug/freeosmemory to trigger garbage collection manually. #539
• fuse: Add configuration MaxCPUs. #546
• fuse: View client log through http proto. #552
• fuse: Support modifying ModifyTime of inode by setAttr. #733
• deployment: Add log map in docker-compose.yaml. #478
• deployment: Introduce nginx to improve compatibility. #534
• test: Add testing script for S3 APIs in python. #514 #595
• monitor: Add detailed vol info in grafana dashboard. #522
• cli: Some new features for CLI tool: bash completions, configuration setting, decommission, diagnose etc. #555 #695

Refactor

• metanode: Accelerate deletion. #582 #600
• datanode: Limit replica number of data partition to at least 3. #587
• datanode: Set if auto-repair by http interface. #672
• metanode datanode fuse: Optimizations on port checking. #543 #531
• metanode datanode: Validate creating partition request from master. #611
• object: Refactor copy object function. #563
• fuse: Replace process of token validation from meta wrapper to client. #498
• fuse: Adjust rlimit config for client. #521

Bug fix

• metanode: When overwriting a file, if the inode in a dentry is updated successfully, ignore unlink and evict inode errors. #500
• metanode: Fix incorrect metrics collection for volume usedGB in monitor system. #503
• metanode: VolStat can correctly update when force update metaPartitions frequently. #537
• metanode: The MaxInodeID of meta partition is not synchronized when recovered from snapshot. #571
• metanode: Free Inodes by raft protocol. #582
• metanode: Painc with deleteMarkedInodes. #597
• object: Fix the parsing issue of the two preconditions of If-Match and If-None-Match. #516
• object: Change the time format in list buckets API to UTC time. #525
• object: Fix xml element of DeleteResult. #532
• object: Empty result in list operation when run in parallel with delete. #509
• object: Change from hard link to soft link in CopyObject action. #563
• object: Solved parallel-safety issue; Clean up useless data on failure in upload part. #553
• object: Fixed a problem in listing multipart uploads. #595
• object: Solve the problem that back-end report “NotExistErr” error when uploading files with the same key in parallel. #685
• fuse: Evict inode cache when dealing with forget. #523

Document

• Update related documents of ObjectNode. #554
• Added user and CLI introduction, synchronize documentation according to the latest code. #564
• Added F&Q section. #573

Release v2.0.0 - 2020/04/10

Feature

• Multi-Zone replication & Create volume on a specified zone. #407 #416
• Support token authentication for readwrite-mount & readonly-mount of fuse client. #435
• A command line tool for cluster operations. #441
• Implemented user security and authorization system to improve resource access control. #441
• Implemented extend attributes (xattr) for metadata and posix-compatible file system interface (mountable client). #441

Enhancement

Object storage related

• Support folder operations in S3 APIs. #450
• Reduce blocking under concurrency. #458
• Implemented more Amazon S3-compatible object storage interfaces to improve compatibility. #441

Master related

• Make replicas of the data partition a specific option when creating volume. #377
• If meta node reaches threshold,set meta partition status to readonly. #411
• Add cluster status API. #457

MetaNode related

• Checks only file type instead of the whole mode. #381

Fuse related

• Make followerRead a client specific option. #382
• Support command line argument for fuse client. #418
• Introduce disable dentry-cache to client option. #453
• Filter target meta partitions in batch iget. #472

Others

• Yum tool for deploying CubeFS cluster. #385

Bug fix

• Fix the signature algorithm issues. #369 #476
• Avoid inode unlink due to net error. #402
• A map structure locked during serialization. #413
• Wait for data sync in close syscall. #419
• Fix empty result on list objects. #433
• Set lookup valid duration for newly created file. #437
• Fix iget error due to metapartition split. #446
• Fix mount fail when volume is full. #453
• Fix offline strategy for raft peers of data partition and meta partition. #467

Document

• Add guide for running CubeFS by yum tools. #386
• Update FUSE client mount options. #439
• Add documentation for client token. #449

Release v1.5.1 - 2020/01/19

Enhancement

• Support building docker image that contains both cfs-server and cfs-client. #353

Bug fix

• Only one replica of meta partition can be taken offline at the same time. #345
• Check if server port is open before raft leader change. #348
• Solved several issue in signature algorithm version 2. #357
• Solved the issue related to copying files across folder through object storage interface. #361

Refactoring

• Update dashboard configuration of grafana. #347
• Unified the configuration of master address and listening port in documentation. #362

Document

• Added benchmark data and guidelines for deploying CubeFS cluster with Helm in README file. #350

Release v1.5.0 - 2020/01/08

Feature

• Add a general Authentication & Authorization framework for CubeFS. commit
• Object storage interface. Add ObjectNode to provide S3-compatible APIs. commit

Enhancement

• Check disk path size in run-docker script. commit
• Support building CubeFS docker image that contains cfs-server and cfs-client. commit
• Add go test in docker-compose. commit
• Add authorization to master api getVol. commit
• Support building under the Darwin(Apple MacOS) and Microsoft Windows operating system environment. commit

Bug fix

• Set DataNode disk size in docker script to be compatible with lower version of df. commit
• The reservedSpace parameter invalid when gt 30GB. commit
• The mtime of parent inode does not change when create or delete a dentry. commit
• MetaNode opResponseLoadPartition removes duplicate locks. commit
• MetaNode leak memory on DeleteMetaPartition operator. commit

Refactoring

• Change configuration file of Master daemon. commit
• Remove unnecessary function in raft store. commit
• Change metaNode loadSnapshotSign command about inodeCount and dentryCount. commit
• Rename rack to cell. commit
• Improve meta partition replicas verification by MaxInode and DentryCnt. commit
• Change parameter name in MetaNode configuration file. commit
• Using current applyID to replace snapshot applyID in MetaNode LoadMetaPartition response action. commit
• Delete data partition and meta partition synchronously. commit

Document

• Add design and user guide document for AuthNode. commit
• Update configuration file sample in document. commit
• Add design and user guide document for object subsystem. commit
• Add IO, small file and metadata performance benchmark data to document. commit

Release v1.4.0 - 2019/11/13

Feature

• Datanode : support read from follower and if packet is tinyExtent,then do write it once commit
• Vol add followerRead field to support reading data from foll owner commit
• Support read from raft follower instead of just leader commit
• Support to modify whether vol supports reading data from a replica commit
• Introduce read and write iops rate limit commit
• Add metrics commit

Enhancement

• If vol has been marked deleted,data partitions, meta partition information reported by heartbeat will no longer be accepted commit
• Use static ip for meta and data nodes commit
• Improve debug environment using docker commit
• Support custom meta node reserved memory commit
• Data partition and meta partition must have three replicas except reducing replicas to 2 commit
• Adjust demo config parameters commit
• Update grafana dashbord for disk error metric commit

Bug fix

• OpFollowerRead if read eof,return error commit
• Get follower read option in init commit
• Stream traverse process never gets triggered in some situation commit
• Check LoadConfigFile before starting daemon commit
• Return error from function LoadConfigFile to the caller commit
• ExtentStorage engine :autoComputeCrc compute crc error commit
• Clean up async delete process of metanode commit
• Set default port to non-system reserved port commit

Refactoring

• Leader change not warning on raft commit
• Remove go module files for now commit
• Clean up response of get all inodes info commit
• Master, DataNode and MetaNode Fix dp or mp offline process commit
• Use AddNodeWithPort replace AddNode,and delete AddNode API commit
• Delete reserved space on DataNode config file commit
• Refine labels of the disk error metric commit
• Optimize auto compute crc commit

Document

• Add use cases commit

Release v1.3.0 - 2019/09/12

Feature

• Introduce writecache mount option. commit
• Introduce keepcache mount option. commit
• Add admin API for get all meta parititons under vol commit
• Support for truncating raft log. commit
• Dynamiclly reduce the num of replicas for vol. commit
• The specified number of replica num is supported when creating vol. commit
• Feature: daemonize server commit
• Support log module change loglevel on line. commit

Enhancement

• Extent_store LoadTinyDeleteFileOffset return s.baseTinyDeleteOffset. commit
• Enable async read by default. commit
• Improve log message details for clientv2. commit
• Compatible with string when get bool config. commit
• Add performance tracepoint for clientv2. commit
• Align out message buffer size with max read size. commit
• For splitting meta partition,updating meta partition and creating new meta partition are persisted within a single transaction. commit
• If metanode used memory is full,then the partition must set to readonly. commit
• Set report time to now after creating data partition. commit
• Set writeDeadLineTime to one minute,avoid metanode gc reset conn which snapshot used as much as possible. commit
• Add raft monitor. commit
• If the creation of a data partition fails, the successfully created replica is deleted. commit
• If the creation of a meta partition fails, the successfully created replica is deleted. commit
• Add unit test case. commit
• Passing create data partition type to datanode. commit
• If create dp is normal,must start Raft else backend start raft. commit
• The tickInterval and electionTick support reading from a configuration file commit

Bugfix

• Fix: add del vol step after ltptest in travis-ci test script. commit
• Clientv2 file handle memory leak. commit
• Redirect stderr to an output file in daemon. commit
• Exclude data partition only when connection refused. commit
• When delete DataParittion,the forwardToLeader mayBe painc. commit
• Metanode load error mayme shield. commit
• Truncate raft corrupt data. commit
• When meta node memory usage arrive threshold, split meta partition occurred dead lock. commit
• SplitMetaPartition race lock with updateViewCache. commit
• After the vol is created and before the heartbeat report, the status of the data partition is set to read only after the check dp operation is performed. commit
• When disk error,the raft cannot start on new data server first. commit
• OpDecommissionDataPartition delete dataPartition on new server. commit
• Datanode may be painc. commit
• Datanode auto compute crc. commit
• DataNode: when dataPartition load,if applyId ==0 ,then start Raft. commit
• The reported data partition usage decreased, and the statistical usage did not decrease accordingly. commit
• Docker metanode.cfg add totalMem parameter. commit
• Datanode deadlock on deletePartition. commit
• DataNode may be painc. commit
• Exclude dir inode in the orphan list. commit
• Evict inode cache after successful deletion. commit
• The actual reduction in the number of replicas exceeds the expected reduction in the number of replicas. commit
• Compatible with old heartbeat mode, old heartbeat mode does not report volname. commit
• Metanode mistakenly delete empty dir inode. commit
• Treat ddelete not exist error as successful. commit
• Fuse directIO read size can exceeds buffer size. commit
• Fix Datanode retain RaftLog commit
• Fix: Datanode: when tinyExtentRepair auto repair,it has been commit
• Fix: Storage :when write tinyExtent,if offset!=e.datasize,return error commit
• The buf is not reset and the old data is repeatedly written to the ump warning log, resulting in the overall performance degradation of the cluster commit
• DataPartitionMap occurred error which is concurrent map iteration and map write commit
• Client gets stale file size if streamer is auto evicted. commit
• Update export init for consul register commit
• Err is shadowed in server main function commit
• If master only create DataPartitionCnt is 10,then cannot mount commit
• Update export init for consul register commit
• DataPartition disk error ,not recvoery raft log on new datanode commit
• Datanode register hang bug commit

Refactoring

• Sdk When creating a datapartition, select the datapartition retry strategy. commit
• Refactoring SDK: when write datapartition num greater 10 ,then trust master. commit
• When disk error,the datapartition recover only recover avali data on tinyExtent. commit
• Decommission Meta or Data Partition must sync response to master. commit
• Synchronized decommission the data partition. commit
• Datanode api /partition add raftStatus. commit
• StartRaftLoggingSchedule not use goroutine. commit
• Metanode must config totalMem. commit
• Sync tinyDeleteExtent time change to 1 days. commit
• Change partitionId varliable to partitionID. commit
• Datanode delete SnapshotFile Pool. commit
• Exporter add ump. commit
• DataNode: compatible old dataPartition Meta info. commit
• Add ltptest log. commit
• Docker metanode config.json change totalMem to 6GB. commit
• Datanode create dataPartition select disk function change. commit
• Datanode start must start StartRaftLoggingSchedule func. commit
• Master create vol min default DataPartition set to 10. commit
• Docker: run ltptest print errorinfo. commit
• When load dp,if dp status is normal,then start raft ,else wait snapshot has recover. commit
• Datanode delete unused func. commit
• If not config warnLogDir,then donnot write umplog to disk. commit
• DataNode: if not config raftDir,then not start server. commit
• Add log on metanode delete extent. commit
• Doc :delete warnLogDir config. commit
• Keep mount point in error stataus when client is killed commit
• Log checkroration checkTime change to 1 second commit
• Metanode change deleteDentry or deleteInode api to log.LogDebugf commit
• Increase the judgment condition of disk error,add syscall.EROFS commit
• Sync code from git.jd.com/cubefs/cubefs commit

Release v1.2.1 - 2019/07/19

Enhancement

• When datanode or metanode start,must compare partition with master and gofmt project. commit
• Datanode add backend check disk status func. commit

Bugfix

• Update rocksdb build depends. commit
• Metanode panic when deleting meta partition. commit
• When raft member become leader,must apply from appliyID to commitID. commit

Release v1.2.0 - 2019/07/16

Feature

• Clientv2: add dentry cache. commit
• Vendor: introduce jacobsa daemonize package. commit
• Client: start client as a daemon. commit
• Clientv2: daemonize client.commit

Enhancement:

• Update docker helper script. commit
• Data sdk: use ip address instead of partition id to rule out unavailable data partitions. commit
• Add monitor add prometheus, grafana in docker helper script.commit
• Add GOPATH check in build.sh. commit
• Integrated rocksdb compilation. commit
• Update makefile. commit
• Docs: start client as a daemon. commit
• Update docs: add make build in readme and docs. commit
• Use one thread to send and recive from follower on primary backup replication protocol .commit
• Change random write raft serialize not use json. commit

Bug fix:

• Fix: when datanode is killed, maxBaseExtentId is corrupted.commit
• Once vol is created successfully, the state of the data partition it has should be writable. commit
• Metanode: fix create dentry exist error. commit
• Add privileged for docker-compose server node. commit
• The retain space field in config file is not allowed to be larger than the default value. commit
• Fix: when data node gets a disk error, the disk status does not changed. commit
• Fix: build error. commit
• Fix build with lua bug. commit
• Check whether vol is valid. If there is no meta partition with start being. commit
• Build: update build.sh for user permission. commit
• Clientv2: fix to get the latest file size. commit
• When create dp failed,then Check all dp host if the host is alive. commit
• Fix bug: when read tinyDeleteFile return eof error. commit
• Build: fix multi users build bug. [commit](https://github.com/cubefs/cubefs/commit/a885777b9da97957cc1eddfee102be3707763b64[commit](https://github.com/cubefs/cubefs/commit/a885777b9da97957cc1eddfee102be3707763b64)
• Build: fix no root user permission bug. commit
• Init replicas info from create data partition request. commit

Change/Refactoring

• Add cfs-base dockerfile. commit
• Specification code and metanode config file. commit
• Add go mod. commit
• Docs: update according to recent changes. commit
• If warnLogDir is not specified in the config files, server or client will return an error. commit
• Rename RestSize to ReservedSize. commit
• Print more detailed error information when server starts failed. commit
• Remove docker client security opt. commit
• Add disk config detailed description. commit
• By default create 10 data partitions and 3 meta partitions when creating volume. commit
• Doc: add clarification that resource manager is AKA master. commit
• Normalize exporter cluster name. commit
• Set the value of rack for data node to default. commit
• Refine decommission disk. commit
• Change document about metanode config. commit
• Doc: update docker helper section. commit
• Update quick-start-guide.rst. commit
• Change dataPartition IntervalToUpdateReplica to 600 seconds. commit
• Add third-party directory. commit
• Write data partition decommission url message to log. commit
• Go fmt project. commit
• Data sdk: check amount of writable data partitions when mount. commit
• Delete third-party directory. commit
• Update docs: remove server and client. commit
• Pass the hosts to data node when creating the data partition. commit
• Warn log message unicode encoding is converted to utf8. commit
• Ignore generated build files. commit
• Build: update build.sh and Makefile. commit
• Change packet attr func and ReadTinyDelete impl log. commit
• Add log when read tinyDeleteFile failed. commit
• Add log when streamRead and ExtentRepairRead. commit
• Remove unused source code. commit
• Build: update build.sh. commit
• Update README.md. commit
• Build: remove client unused depends. commit

Release v1.1.1 - 2019/06/11

Feature

• Client: enable support to fifo and socket file types. commit
• Clientv2: use jacobsa fuse package instead of bazil. commit
• Docker: introduce docker compose to create a local testing CubeFS cluster. commit

Bugfix

• Meta: update atime in inode get operation. [metanode] commit
• Fix: potential panic if send returns nil resp with no error. [client] commit
• Fix: raft election takes a long time and timeout; issue a panic if raft server is down. [raft] commit
• Fix: potential deadlock if applyHandler. [master] commit
• Fix: put vol to cache after it is persistent. [master] commit
• Fix: partition is nil when apply remove raft node. [datanode] commit
• Fix: metanode painc. [metanode] commit
• Fix: panic when pprof does not start. commit

Enhancement

• Sdk: retry if mount failed in case master is unavailable temporarily. commit
• Build: add verbose build info. commit
• Master: introduce data partition over-provision. commit
• Master: reserve writable data partition amount according to capacity instead of a const. commit
• Monitor: Use UMP performance monitor if exporter is not enabled. commit

Release v1.1.0 - 2019/05/07

Change / Refactoring

• Rename the repository from cfs to cubefs.
• Use own errors module instead of juju errors due to license incompatibility.
• Metanode: Change not raft leader error to tryOtherAddr error.

Bugfix

• Master: Partition recovered but the status not changed.
• Datanode: Report to client with proto.OptryAgain resultcode when datapartition does not exsit.
• Raft: A member must apply playback from old apply id to commit id after elected as leader.
• Metanode: generate identical inode number under extreme conditions. commit

Enhancement

• Master: Add ump warn packet.
• Master: Remove redundant calling of loadMetaData method.
• Master: Reload meta data after leader changed.
• Master: make dataPartition disk Path persistent. commit
• Master: Volume creation supports specifying the amount of meta partitions. commit
• Metanode: Add totalMem in configFile.
• Datanode: Change default disk reserved space.
• Datanode: Add volname in heartbeat report.
• Raft: Use raft.ErrNotLeader instead of ErrNotLeader.
• Client: Create a dummy node instance if inode does not exist.
• Client: Add UMP monitor alarms for read/write/fsync errors.
• Client: Suppress some error messages. commit
• Log: Automatically create subdirectory under the log directory.

Release v1.0.0 - 2019/04/02

The initial release version.