Zhihao Cheng [Mon, 11 Nov 2024 09:01:09 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Remove deleted nodes from valid node tree
This is the 2/12 step of rebuilding. Traverse nodes from del_inos and
del_dents trees, remove inode nodes and dentry nodes with smaller sqnum
from valid trees.
This step handles deleting case, for example, file A is deleted, deleted
inode node and deleted dentry node are written, if we ignore the deleted
nodes, file A can be recovered after rebuilding because undeleted inode
node and undeleted dentry node can be scanned. There's an exception, if
deleted inode node and deleted dentry node are reclaimed(by gc) after
deletion, file A is recovered. UBIFS rebuild_fs cannot solve it, because
the real existence information of nodes depends on TNC, but TNC should
not be depended for UBIFS rebuild_fs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:08 +0000 (17:01 +0800)]
fsck.ubifs: Add rebuilding filesystem support
Add rebuilding filesystem support. This is the 1/12 step of rebuilding.
Collect files, valid inode nodes, deleted inode nodes, valid dentry
nodes and deleted dentry nodes in kinds of trees by scanning nodes from
flash. Corrupted nodes(eg. incorrect crc, bad inode size, bad dentry
name length, etc.) are dropped during scanning. Larger sqnum node is
picked when more than 1 nodes with same index.
In this step, trun node and data nodes are put into corresponding file,
inode/dentry nodes are put into four trees: valid_inos(nlink != 0),
del_inos(nlink is 0), valid_dents(inum != 0), del_dents(inum is 0).
Next step will process above four trees to deal deletion situations.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:07 +0000 (17:01 +0800)]
fsck.ubifs: Add file organization realization
In order to check the consistency of each file and the reachability of
the whole dentry tree, fsck orginizes all nodes into files. And the
final recovered file(xattr is treated as a file) is organized as:
(rbtree, inum indexed)
/ \
file1 file2
/ \
file3 file4
file {
inode node // each file has 1 inode node
dentry (sub rb_tree, sqnum indexed) // '/' has no dentries,
// otherwise at least 1
// dentry is required.
trun node // the newest one truncation node
data (sub rb_tree, block number indexed) // Each file may have 0
// or many data nodes
xattrs (sub rb_tree, inum indexed) // Each file may have 0 or
// many xattr files
}
Each file from file rb_tree is constructed by scanning nodes(eg. inode,
dentry, etc.) from the TNC or the UBI volume. File's xattrs will be
initialized in subsequent steps.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:06 +0000 (17:01 +0800)]
fsck.ubifs: Add node parsing functions
Add parsing functions for each type of nodes, which will be used for
checking the validity of raw node data while reading from TNC or
scanning from UBIFS logical erase block.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:05 +0000 (17:01 +0800)]
fsck.ubifs: Load filesystem information from UBI volume
Load filesystem information from UBI volume (Similar to UBIFS mounting
process), initialize kinds of buffers and read superblock. This is the
base step for both fsck and rebuild_fs. Subsequent pacthes will complete
this step by adding more steps(eg. read master, replay journal, etc.)
which are only used in fsck.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:04 +0000 (17:01 +0800)]
fsck.ubifs: Distinguish reasons when certain failures happen
Read failure caused by scanning corrupted data or invalid data members
should be identified, because fsck can handle it. Updating lp failure
caused by bad space statistics should be identified too, because fsck
can handle it.
Add eight callback functions to implement it for fsck:
1. set_failure_reason_callback: Record failure reasons when reading or
parsing node failed, there are four reasons:
a. FR_DATA_CORRUPTED: scanning corrupted data or invalid nodes found
b. FR_TNC_CORRUPTED: invalid index nodes
c. FR_LPT_CORRUPTED: invalid pnode/nnode
d. FR_LPT_INCORRECT: invalid space statistics or invalid LEB properties
2. get_failure_reason_callback: get failure reasons
3. clear_failure_reason_callback: Clear the error which is caused by
above reasons.
4. test_and_clear_failure_reason_callback: Check and clear the error
which is caused by above reasons, if so, fsck will handle it
according to specific situation.
For example, fsck will drop data node rather than fails to return
when reading failure is caused by DATA_CORRUPTED.
For another example, journal replaying will continue rather than
fails to return if updating lpt failure is caused by LPT_CORRUPTED.
5. set_lpt_invalid_callback: Set the invalid lpt status
6. test_lpt_valid_callback: Check whether the lpt is corrupted/incorrect,
it should be invoked before updating lp, if lpt status is invalid,
returns false (which means that caller should skip updating lp, because
updating lp could trigger assertion failed in ubifs_change_lp).
7. can_ignore_failure_callback: Check whether the failure can be
ignored, some inconsistent errors won't affect the fsck process,
for example wrong space statistics can be fixed after traversing
TNC, so failures caused by incorrect space statistics can be ignored.
8. handle_failure_callback: Check whether the failure can be handled,
some inconsistent errors could be fixed by fsck, we have fix_problem
to do that, but UBIFS needs a callback function to invoke it in common
libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:03 +0000 (17:01 +0800)]
fsck.ubifs: Add inconsistent problem handling asking function
There are four dimensions to define the type of inconsistent problems:
1. fixable: Some inconsistent problems can't be fixed, for example
corrupted superblock. Un-fixable problem will abort program.
2. must fix: Some inconsistent problems can be ignored(eg. incorrect
isize), but some are not(eg. corrupted TNC), which will affect the
subsequent fsck steps.
3. drop data: Some fixing methods will drop user data, which is
unacceptable for safe mode. If it happens, fsck will be aborted.
4. need rebuild: Some inconsistent problems depends on rebuilding
filesystem to be fixed(eg. corrupted master node, corrupted TNC).
Define an asking function to handle above kinds of inconsistent problems.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:02 +0000 (17:01 +0800)]
fsck.ubifs: Add fsck support
Add basic process code for fsck.ubifs. There are following modes for fsck:
1. normal mode: Check the filesystem, ask user whether or not to fix
the problem as long as inconsistent data is found during fs checking.
2. safe mode: Check and safely repair the filesystem, if there are any
data dropping operations needed by fixing, fsck will fail.
3. danger mode: Answer 'yes' to all questions. There two sub modes:
a) Check and repair the filesystem according to TNC, data dropping
will be reported. If TNC/master/log is corrupted, fsck will fail.
b) Check and forcedly repair the filesystem according to TNC, turns
to rebuild filesystem if TNC/master/log is corrupted. Always make
fsck succeed.
4. check mode: Make no changes to the filesystem, only check the
filesystem.
5. rebuild mode: Scan entire UBI volume to find all nodes, and rebuild
filesystem, always make fsck success.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:00 +0000 (17:01 +0800)]
ubifs-utils: Replace ubifs related source code with linux kernel implementation
Replace ubifs related source code with the implementation of linux kernel.
It makes userspace implementation be same with linux kernel, then
fsck.ubifs can resuse the code.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:00:57 +0000 (17:00 +0800)]
ubifs-utils: Adapt gc subsystem in libubifs
Adapt gc subsystem(find.c, gc.c, scan.c) in libubifs, compared with
linux kernel implementations:
1. Adapt print_hex_dump based on implementations in hexdump.c.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:00:56 +0000 (17:00 +0800)]
ubifs-utils: Adapt orphan.c in libubifs
Adapt orphan.c in libubifs, compared with linux kernel implementations:
1. Keep the commit related implementations, because do_commit depends
on these functions which will be invoked in fsck.
2. Keep the orphan replaying implementations, because fsck needs them.
3. Other implementations are removed which won't be used in mkfs/fsck.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:00:55 +0000 (17:00 +0800)]
ubifs-utils: Adapt misc.h in libubifs
Adapt misc.h in libubifs, compared with linux kernel implementations:
1. Remove some functions(eg. ubifs_compr_name, ubifs_wake_up_bgt) which
won't be used in fsck/mkfs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:00:54 +0000 (17:00 +0800)]
ubifs-utils: Adapt master.c in libubifs
Adapt master.c in libubifs, compared with linux kernel implementations:
1. Remove authentication related implementations
(eg. mst_node_check_hash), authentication is not supported in fsck
for now.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:10 +0000 (16:37 +0800)]
ubifs-utils: Adapt key.h in libubifs
Adapt key.h in libubifs, compared with linux kernel implementations:
1. Add '__unused' modifier for unused parameters to avoid compiling
warnings.
2. Remove some functions(eg. lowest_dent_key, dent_key_init_flash)
which won't be used in fsck/mkfs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:09 +0000 (16:37 +0800)]
ubifs-utils: Adapt debug subsystem in libubifs
Adapt debug subsystem(debug.c, debug.h) in libubifs, compared with
linux kernel implementations:
1. Only keep the disk data and space statistics dumping implementations,
dbg_walk_index and add_size which are used by fsck, other debuging
related implementations and sysfs interfaces are removed, because
fsck will check fs in another way.
2. Change the behavior of ubifs_assert_failed(), make filesystem
readonly when assertion is failed.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:08 +0000 (16:37 +0800)]
ubifs-utils: Adapt commit.c in libubifs
Adapt commit.c in libubifs, compared with linux kernel implementations:
1. Remove debug related implementations(eg. dbg_check_old_index), debug
functions are not needed by fsck, because fsck will check fs in
another way.
2. Remove ubifs background committing related implementations, there
will be no background ubifs threads in fsck.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:07 +0000 (16:37 +0800)]
ubifs-utils: Adapt budget.c in libubifs
Adapt budget.c in libubifs, compared with linux kernel implementations:
1. Remove writeback related functions, there are no dirty pages/inodes
for UBIFS in userspace process.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:06 +0000 (16:37 +0800)]
ubifs-utils: Adapt journal.c in libubifs
Adapt journal.c in libubifs, compared with linux kernel implementations:
1. Remove all ubifs_jnl_XXX functions. Only keep the basic space
reservation code, fsck will add new functions for journaling
operations without using linux in-memory inode/dentry.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:05 +0000 (16:37 +0800)]
ubifs-utils: Adapt dir.c in libubifs
Adapt dir.c in libubifs, compared with linux kernel implementations:
1. Remove all functions. Only keep an empty source file, fsck will
add new functions for mkdir/link operations without using linux
in-memory inode/dentry.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:04 +0000 (16:37 +0800)]
ubifs-utils: Adapt auth.c in libubifs
Adapt auth.c in libubifs, compared with linux kernel implementations:
1. Only keep implementations used by mkfs, other implementations
are removed.
2. Adapt functions based on implementations in common/sign.c.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:03 +0000 (16:37 +0800)]
ubifs-utils: Adapt sb.c in libubifs
Adapt sb.c in libubifs, compared with linux kernel implementations:
1. Remove authentication related implementations
(eg. authenticate_sb_node), authentication is not supported in fsck
for now.
2. Remove some functions(eg. create_default_filesystem) which won't be
used in fsck/mkfs.
3. Modify ubifs_read_superblock(), remove some assignments which won't
be used in mkfs/fsck.
4. Apapt fixup_leb to ignore %-EBADMSG, subsequent steps will check
all areas carefully.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:02 +0000 (16:37 +0800)]
ubifs-utils: Adapt recovery subsystem in libubifs
Adapt recovery subsystem(replay.c, recovery.c) in libubifs, compared with
linux kernel implementations:
1. Remove authentication related implementations
(eg. authenticate_sleb_hash), authentication is not supported in fsck
for now.
2. Add explicit type conversions(const char *) to avoid compiling
warnings.
3. Replace implementations of inode_fix_size() with ubifs_assert(0),
authentication is not supported in fsck, so this function won't
be invoked.
4. Remove unused ubifs_clean_lebs() and ubifs_write_rcvrd_mst_node().
5. Adapt fix_unclean_leb/recover_head/fix_size_in_place to ignore
%-EBADMSG, subsequent steps will check nodes in lpt/main area
carefully.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:01 +0000 (16:37 +0800)]
ubifs-utils: Adapt log.c in libubifs
Adapt log.c in libubifs, compared with linux kernel implementations:
1. Remove debug related implementations(eg. dbg_check_bud_bytes), debug
functions are not needed by fsck, because fsck will check fs in
another way.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:00 +0000 (16:37 +0800)]
ubifs-utils: Adapt tnc subsystem in libubifs
Adapt tnc subsystem(tnc.c,tnc_misc.c,tnc_commit.c) in libubifs, compared
with linux kernel implementations:
1. Remove debug related functions(eg. dbg_check_inode_size), debug
functions are not needed by fsck, because fsck will check fs in
another way.
2. Remove some functions(eg. ubifs_tnc_bulk_read) which won't be used
in fsck/mkfs.
3. Adapt tnc_delete and ubifs_search_zbranch to handle empty TNC case,
which could happen in fsck.
4. Don't skip checking the length of non-leaf index node's branch in
read_znode.
5. Adapt try_read_node to ignore %-EBADMSG, subsequent steps will check
nodes carefully.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:59 +0000 (16:36 +0800)]
ubifs-utils: Adapt lpt subsystem in libubifs
Adapt lpt subsystem(lpt.c,lprops.c,lpt_commit.c) in libubifs, compared
with linux kernel implementations:
1. Remove debug related functions(eg. dbg_chk_lpt_sz, dbg_chk_pnode),
some of debug functions are not needed by fsck, because fsck will
check fs in another way.
2. Remove some functions(eg. ubifs_create_dflt_lpt) which won't be used
in fsck/mkfs.
3. Adapt do_calc_lpt_geom() to mkfs/fsck situations.
4. Adapt calc_dflt_lpt_geom to support the mkfs tool, and export it.
5. Adapt ubifs_create_lpt() according to create_lpt(mkfs), make sure that
the height of created lpt is right.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:58 +0000 (16:36 +0800)]
ubifs-utils: Adapt io.c in libubifs
Adapt io.c in libubifs, compared with linux kernel implementations:
1. Modify io related functions(eg. ubifs_leb_read/ubifs_leb_write,
etc.), adapt them with userspace io functions lseek/read/write.
2. Remove some functions(eg. record_magic_error, ubifs_bg_wbufs_sync)
which won't be used in fsck/mkfs.
3. Replce ubifs_errc with ubifs_err, because there will be no SB_SILENT
options in mkfs/fsck.
4. Initiate wbuf->size as c->max_write_size.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:57 +0000 (16:36 +0800)]
ubifs-utils: Adapt super.c in libubifs
Adapt super.c in libubifs, compared with linux kernel implementations:
1. It contains all definitions in common/super.c(Message printing
functions are replaced with linux kernel styles).
2. Remove some functions(eg. ubifs_iget, ubifs_dirty_inode) which won't
be used in fsck/mkfs.
3. Remove unused variables initialization in some functions(eg.
init_constants_early, init_constants_sb).
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:56 +0000 (16:36 +0800)]
ubifs-utils: Adapt ubifs header file in libubifs
Adapt ubifs header file in libubifs, compared with linux kernel
implementations:
1. New header file contains all definitions in common/ubifs.h
2. Remove some structures(eg. ubifs_mount_opts, bu_info) and
functions(eg. ubifs_sync_wbufs_by_inode, ubifs_jnl_XXX) which
won't be used in fsck/mkfs.
3. Modify some authentication related functions, keep functions
that mkfs may need, other functions are defined as empty.
4. Move ubifs message printing functions(ubifs_err/warn/msg) which
are originally defined in misc.c into ubifs.h.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:51 +0000 (16:36 +0800)]
ubifs-utils: Add implementations for linux kernel printing functions
Add implementations for linux kernel printing functions, because these
functions(eg. pr_debug, pr_err, etc.) are widely used in UBIFS linux
kernel libs. No need to define multiple levels in dbg_msg for debuging,
just replace dbg_msg with pr_debug. Now, there are five levels of
printing messages:
0 - none
1 - error message
2 - warning message [default]
3 - notice message
4 - debug message
This is a preparation for replacing implementation of UBIFS utils with
linux kernel libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:49 +0000 (16:36 +0800)]
ubifs-utils: Add rwsem implementations
Add rwsem implementations, because there are some rwsems
(eg. c->commit_sem) used in UBIFS linux kernel libs.
The rwsem is implemented based on pthread mutex.
This is a preparation for replacing implementation of UBIFS utils with
linux kernel libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:48 +0000 (16:36 +0800)]
ubifs-utils: Add mutexlock implementations
Add mutexlock implementations, because there are some mutexlocks
(eg. c->tnc_mutex, c->log_mutex) used in UBIFS linux kernel libs.
The mutexlock is implemented based on pthread mutex.
This is a preparation for replacing implementation of UBIFS utils with
linux kernel libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:47 +0000 (16:36 +0800)]
ubifs-utils: Add spinlock implementations
Add spinlock implementations, because there are some spinlocks
(eg. c->cnt_lock, c->buds_lock) used in UBIFS linux kernel libs.
The spinlock is implemented based on pthread mutex.
This is a preparation for replacing implementation of UBIFS utils with
linux kernel libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:42 +0000 (16:36 +0800)]
ubifs-utils: Add linux type definitions
Add linux type definitions, because there are many types
(eg. u8/u16/u64) used in UBIFS linux kernel libs. Besides
move type conversions (eg. cpu_to_le16, cpu_to_le32, etc.)
into type definitions header file.
This is a preparation for replacing implementation of UBIFS utils with
linux kernel libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Add compiler attributes implementations, such as __packed, __unused,
__const, etc., which could be used in linux kernel libs. Besides, change
some existing attributes into linux kernel style.
This is a preparation for replacing implementation of UBIFS utils with
linux kernel libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:40 +0000 (16:36 +0800)]
ubifs-utils: Import UBIFS libs from linux kernel
Import UBIFS libs from linux kernel. Next patches will replace ubifs
related source code with implementation of linux kernel, which makes
userspace implementation be same with linux kernel, then fsck.ubifs
can resuse the code.
Notice: lpt.c is modified with [1] applied. ubifs.h and orphan.c are
modified with [2] applied, journal.c is modified with [3] reverted(
because fsck runs in a single thread, so waitqueue is not needed to
be implemented in userspace.).
[1] https://lore.kernel.org/linux-mtd/20231228014112.2836317-13-chengzhihao1@huawei.com/
[2] https://lore.kernel.org/linux-mtd/20240410073751.2522830-1-chengzhihao1@huawei.com/
[3] https://lore.kernel.org/linux-mtd/20240122063103.359501-1-chengzhihao1@huawei.com/
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:38 +0000 (16:36 +0800)]
ubifs-utils: Add 'dev_fd' and 'libubi' into 'ubifs_info' structure
Embed new members 'dev_fd' and 'libubi' into ubifs_info structure, so
that global variable 'ubi', 'out_fd' and 'out_ubi' could be removed
from mkfs.ubifs.c. Besides, add parameter in check_volume_empty().
Next patch will extract UBI opening/closing/check_volume_empty functions
into a new source file, these functions will be used in fsck.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:37 +0000 (16:36 +0800)]
ubifs-utils: Add 'dev_name' into 'ubifs_info' structure
Embed new member 'dev_name' into 'ubifs_info' structure, then global
variable 'output' can be removed from mkfs.ubifs.c. Next patches will
import UBIFS libs from linux kernel, which could print messages that
contain device information, so this patch can distinguish different
devices according to messages.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:36 +0000 (16:36 +0800)]
ubifs-utils: Move 'debug_level' into ubifs_info structure
Embed new member 'debug_level' into 'ubifs_info' structure, then global
variable 'debug_level' can be removed from mkfs.ubifs.c. Next patches
will import UBIFS libs from linux kernel, which could print messages
with different levels, then 'debug_level' can be used to control which
level messages should be printed.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:35 +0000 (16:36 +0800)]
ubifs-utils: Clean up error message printing functions
Functions 'err_msg' and 'sys_err_msg' are almost same with 'errmsg' and
'sys_errmsg', since 'errmsg' and 'sys_errmsg' can print programe name,
so replace error message printing functions (ubifs-utils) with common
lib functions(include/common.h).
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:34 +0000 (16:36 +0800)]
ubifs-utils: Define PROGRAM_NAME as variable
PROGRAM_NAME is defined as a const string "mkfs.ubifs", which won't be
suitable for fsck.ubifs. Add 'program_name' member in ubifs_info
structure, define PROGRAM_NAME as ubifs_info.program_name. Then, error
messages will display right program name if fsck.ubifs is supported.
Besides, add 'program_type' member in ubifs_info structure to identify
which current program type is, without comparing program name.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:33 +0000 (16:36 +0800)]
ubifs-utils: Decouple mkfs.ubifs.h out of other modules
Header file mkfs.ubifs.h is included in other modules(eg. compr.c, lpt.c,
fscrypt.h, sign.c), decouple it out of other modules.
There are two parts in mkfs.ubifs.h:
1. common functions, for example dbg_msg, err_msg and write_leb, move
these functions into common/defs.h and common/ubifs.h.
2. devtable related definations, move them into a new header file
common/devtable.h.
Splitting common functions from mkfs.ubifs.h is also a preparation for
importing libubifs(from linux kernel) to replace current UBIFS libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:32 +0000 (16:36 +0800)]
ubifs-utils: Split common source files from mkfs.ubifs
Split common source files into common dir from mkfs.ubifs, this is a
preparation for importing libubifs(from linux kernel) to replace
current UBIFS libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Currently the read throuput test tries:
- 1 page
- 2 pages
- 1 block (64 or more pages, usually)
But it might be interesting to see how the speed gradually increases,
eg. testing all number of pages from 1 to maybe 16, and then
arbitrarilly 32 and 64.
Let's add a -C parameter to enable this additional test.
The 2-page read/write tests are also moved under this new option.
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Miquel Raynal [Mon, 26 Aug 2024 09:46:28 +0000 (11:46 +0200)]
mtd-tests: flash_speed: Drop read_eraseblock_by_page()
The read_eraseblock_by_2pages() has been generalized so it became
read_eraseblock_by_npages(), but there is no limitation (besides 0)
regarding the number of pages. Hence, drop the _by_page() helper and
replace it with the _by_npages(), using 'npages = 1'.
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Right now there are only 2 pages that may be read continuously, but why
not trying more? At least when the continuous feature is out, this type
of benchmarking will be interesting. In order to facilitate later
additions, lets make this helper more generic and accept a global
'npages' variable as parameter (because this function is called in a
macro, it is simpler like that).
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Miquel Raynal [Mon, 26 Aug 2024 09:46:26 +0000 (11:46 +0200)]
mtd-tests: flash_speed: Clarify the number of pages in each set while measuring
So far speed calculations have only be done 1 page at a time or 2 pages
at a time in a block; so basically all the block was always read because
all blocks are multiple of 2. But in the future, if we want to extend
the number of pages in a single read, the final number of pages actually
read might be less than an erase block size, hence failing the throuput
calculations.
Make the number of pages in a set explicit.
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Miquel Raynal [Mon, 26 Aug 2024 09:46:25 +0000 (11:46 +0200)]
mtd-tests: flash_speed: Drop an apparently useless block
I know no device without a multiple of 2 number of pages in each
block. Even though it might be the case, it is clearly not a big deal
and we don't really care about reading the last page, we are doing a
speed benchmark; so as long as the throughput calculation knows how much
data has been read it's fine. Eitherway, I don't think we ever have
fallen in this block because we would read the content of two pages (so
one past the block) and put it in a page-wide buffer, which would
probably lead to an out-of-bound abort.
Just drop the block.
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Miquel Raynal [Mon, 26 Aug 2024 09:46:24 +0000 (11:46 +0200)]
mtd-tests: nandbiterrs: Add support for testing continuous reads
In order to trigger a continuous read, the user needs to request at
least two pages at the same time. So far the tool would only read single
pages, so let's extend its capabilities to test continuous read output
when the -c option is passed.
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Miquel Raynal [Mon, 26 Aug 2024 09:46:23 +0000 (11:46 +0200)]
mtd-tests: nandbiterrs: Store the chunks size in an intermediate variable
'pagesize' is used for two purposes:
- accessing the size of a page
- getting the size of the test buffer, which happen to be the size of a
page for now.
Use an intermediate variable when getting the size of the test buffer,
as we will later increase its size.
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Miquel Raynal [Mon, 26 Aug 2024 09:46:22 +0000 (11:46 +0200)]
nand-utils: nanddump: Add support for testing continuous reads
In order to trigger a continuous read, the user needs to request more
than one data page. So far the tool would split the length into page
chunks. This is no longer the case when the -C option is passed (-c is
already used for the canonical output style).
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Miquel Raynal [Mon, 26 Aug 2024 09:46:21 +0000 (11:46 +0200)]
nand-utils: nanddump: Explicitely use the page size when relevant
Using bs when skipping the bad sector is abusive as what we want is
using the size of a block and the size of a page. The fact that bs
currently is the size of a page is misleading here, has I intend to make
this amount grow.
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Miquel Raynal [Mon, 26 Aug 2024 09:46:20 +0000 (11:46 +0200)]
nand-utils: nanddump: Use a specific variable for the buffer size
The read buffer size happen to be as big as the bs variable, but this is
going to change. When accessing the buffer size, use a specific variable
instead.
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Fri, 2 Feb 2024 02:38:52 +0000 (10:38 +0800)]
mtd-utils: Extract rbtree implementation to common lib
Current rbtree implementation code is put under jffs utils, extract it
into common lib, and add more rbtree operations(eg. rb_first_postorder,
rb_next_postorder, etc.).
This is a preparation for replacing implementation of UBIFS utils with
linux kernel libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Fri, 2 Feb 2024 02:22:53 +0000 (10:22 +0800)]
mtd-utils: Extract list implementation to common lib and add list_sort support
Current list implementation code is put under jffs utils, extract it into
common lib, and add more list operations(eg. list_move, list_splice, etc.).
Besides, add list sorting support in new source file lib/list_sort.c.
This is a preparation for replacing implementation of UBIFS utils with
linux kernel libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Thu, 22 Feb 2024 12:37:40 +0000 (20:37 +0800)]
mkfs.ubifs: Fix memleak for 'output' in error paths
The 'output' is allocated in get_options(), don't forget to free it
in error paths, move 'output' freeing out of close_target(), which
simplifies the logic of close_target().
Zhihao Cheng [Thu, 22 Feb 2024 12:27:58 +0000 (20:27 +0800)]
mkfs.ubifs: Fix missed closing out_fd
Closing 'out_fd' is missed in handling paths in open_target(), fix it
by adding closing operations before returning.
Fixes: a48340c335dab ("mkfs.ubifs: use libubi to format UBI volume") Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Thu, 22 Feb 2024 12:10:38 +0000 (20:10 +0800)]
mkfs.ubifs: Close libubi in error handling paths
The libubi could be opened in get_options(), don't forget to close it
in error handling paths in main(). Also close libubi in error handling
paths in open_ubi(). To implement that, extract libubi_close() into a
new helper function close_ubi().
Besides, invoke crypto_cleanup() in error handling paths in main().
Fixes: a48340c335dab ("mkfs.ubifs: use libubi to format UBI volume") Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Thu, 9 May 2024 12:30:08 +0000 (20:30 +0800)]
mkfs.ubifs: Fix incorrect dir size calculation in encryption scenario
The size of directory should be the total length of encrypted entry name,
otherwise it could trigger errors while checking filesystem:
dbg_check_filesystem [ubifs]: directory inode 89 size is 352, but
calculated size is 400
Zhihao Cheng [Sun, 4 Feb 2024 02:21:44 +0000 (10:21 +0800)]
mkfs.ubifs: Initialize 'cipher_name' as NULL
Variable 'cipher_name' is defined on stack without initialization, when
an user invokes mkfs with '--key' and without '-C', 'cipher_name' is a
random value, which could cause init_fscrypt_context() failed to find a
valid cipher.
Fix it by initializing 'cipher_name' as NULL when it is declared.
Fixes: cc4c5e295f546 ("mkfs.ubifs: Enable support for building without crypto") Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Wed, 7 Feb 2024 01:29:43 +0000 (09:29 +0800)]
mkfs.ubifs: Clear direct_write property when closing target
Direct writing is not allowd on an UBI volume, unless the volume is set
with direct_write property. The open_target sets direct_write property,
don't forget to clear direct_write property for UBI volume when closing
target.
Fixes: a48340c335dab ("mkfs.ubifs: use libubi to format UBI volume") Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
PKG_CHECK_MODULES confusingly does not check to see if a static version
of the library is available.
Actually with older glibc, it might fail as lpthread needs to be passed
to linker flags (Libs.private in pc files) which can only be obtained if
--static is passed to pkg-config.
Signed-off-by: Rosen Penev <rosenp@gmail.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Csókás, Bence [Mon, 26 Aug 2024 11:24:59 +0000 (13:24 +0200)]
fs-tests: integck: Refactor: split out common remount logic
remount_tested_fs() and recover_tested_fs() both have
almost the same code for remounting the target FS
RO then RW. Split this sequence into a new
function called remount_ro_rw().
Csókás, Bence [Mon, 26 Aug 2024 11:24:57 +0000 (13:24 +0200)]
fs-tests: integck: Refactor: split out common remount logic
remount_tested_fs() and recover_tested_fs() both have
almost the same code for umount'ing the target FS and
mount'ing it back. Split this sequence into a new
function called umount_and_remount().
Calling ubi_update_start() may fail if exclusive access to the UBI
volume cannot be established by the kernel. This is likely to happen if
an UBI volume is updated directly after creation, because current
versions of (systemd-)udevd run their internal 'blkid' also for UBI
volumes as soon as they appear (trying to read the UBIFS UUID).
There are mainly to options for this case:
1. Don't allow udevd to read UBI volumes at all (means loosing
functionatlity)
2. Wait until udevd has finished accessing a new volume (requires
cooperation with udevd)
3. Simply retry after short time in case of EBUSY
For using option 3, the documentation should state that the concrete
error code can be read via errno (this has always been the case since
libubi has been introduced).
Link: https://groups.google.com/g/swupdate/c/8NVooKjD9oo Signed-off-by: Christian Eggers <ceggers@arri.de> Reviewed-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Ben Hutchings [Fri, 12 Apr 2024 20:55:05 +0000 (22:55 +0200)]
mtd-utils: fectest: Fix time formatting with _TIME_BITS=64 on 32-bit system
fectest.c formats a struct timeval with the assumption that time_t and
suseconds_t are aliases for long, but some 32-bit systems now define
them as long long in order to support timestamps beyond 2038.
As this is an elapsed time and not a timestamp, both fields should be
within the range of long, so cast them to long before formatting
rather than moving to long long.
Signed-off-by: Ben Hutchings <ben.hutchings@mind.be> Reviewed-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
David Oberhollenzer [Sun, 28 Feb 2021 14:18:33 +0000 (15:18 +0100)]
Cleanup: Unify handling of OpenSSL dependency
This commit modifes the handling of the OpenSSL dependency to work the
same as the other optional dependencies, except that the variable
"with_crypto" is used instead of "with_openssl".
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
David Oberhollenzer [Sat, 17 Feb 2024 15:37:56 +0000 (16:37 +0100)]
Make zlib an optional dependency
Now that we have plumbing in place for both jffsX-utils and mkfs.ubifs,
add the missing autoconf and automake changes to allow mtd-utils to be
built without a hard dependency on zlib.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
David Oberhollenzer [Sun, 18 Feb 2024 13:22:08 +0000 (14:22 +0100)]
Make it possible to compile mkfs.ubifs without zlib
This one is a bit trickier than adding WITH_ZLIB ifdefs. Some parts
of the code assume that zlib is always present and have complicated
fallback behavior. Particularly the "favor_lzo" compression method
that uses either zlib or lzo, whichever produces the better result.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
David Oberhollenzer [Sun, 18 Feb 2024 12:52:42 +0000 (13:52 +0100)]
Make it possible to compile jffsX-utils without zlib
The jffsX-utils already have a CONFIG_JFFS2_ZLIB define, but it is
statically defined in a header and not used consistently. This patch
first replaces it with a "WITH_ZLIB" define, provided via automake,
to bring it in line with the already existing "WITH_LZO". Then, the
missing checks and typedefs are added in jffs2reader.c.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
David Oberhollenzer [Sat, 17 Feb 2024 20:13:24 +0000 (21:13 +0100)]
Fix optional compilation of lzo compressors
- Simple fixup in autoconf.ac
- Make compilation of jffsX-utils compr_lzo.c optional. Simply
don't compile it if we built without LZO and remove the place
holders. They are not used anyway if we build without LZO.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
David Oberhollenzer [Sun, 28 Feb 2021 12:50:57 +0000 (13:50 +0100)]
Cleanup handling of optional dependencies
Don't use super pedantic parsing of the argument and work with the
generated variable instead of assigning it to our own and set it
to "check" if not value is assigned. Then search for a dependency
if the with variable is anything other than "no" and fail if it
was set to "yes".
In addition, the WITHOUT_xxx defines are replaced with WITH_xxx defines.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
David Oberhollenzer [Wed, 14 Feb 2024 12:58:36 +0000 (13:58 +0100)]
Add missing autoconf check for AR
After removing libtool, there is no more cross-check for AR. Because
some slightly older versions of autoconf don't have AC_PROG_AR, simply
insert the implementation one-liner.
We already tried bumping the minimum autoconf version once in the past,
but apparently that is an issue for a number of distros.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
David Oberhollenzer [Fri, 15 Dec 2023 11:42:41 +0000 (12:42 +0100)]
mkfs.ubifs: fix xattr scanning for builds with selinux support
mkfs.uibfs can add Selinux xattrs from a labeling file using
libselinux to parse it. The commit that added this feature simply
introduced a separate function, inode_add_selinux_xattr, which is
called instead of inode_add_xattr. If no --selinux argument is
specified for mkfs.ubifs, this is a no-op.
The problem is, that this breaks xattr scanning for any build with
Selinux enabled. The Selinux version is always called and it does
not scan for xattrs on the filesystem, or dispatch to the original.
This commit fixes the xattr scanning behavior. We unconditionally call
both functions (they each have no-op implementations if the feature
is missing) and in the regular xattr scanning code, we skip selinux
attributes, if the --selinux option was given.
Zhihao Cheng [Mon, 13 Nov 2023 09:48:12 +0000 (17:48 +0800)]
ubiattach: introduce need_resv_pool in UBI_IOCATT
The ioctl UBI_IOCATT has been extended with need_resv_pool parameter in
[1].
This parameter is used for deciding whether to reserve PEBs for filling
pool/wl_pool for target ubi device. This parameter will be effective
when fastmap is enabled, which will slow down the frequency of updating
fastmap by filling more free PEBs in pool/wl_pool. See details in [2].
Zhihao Cheng [Mon, 13 Nov 2023 09:48:11 +0000 (17:48 +0800)]
ubiattach: introduce disable_fm in UBI_IOCATT
The ioctl UBI_IOCATT has been extended with disable_fm parameter after
[1].
This parameter is used for disabling fastmap for target ubi device.
If 'disable_fm' is set, ubi doesn't create new fastmap even the module
param 'fm_autoconvert' is set, and existed old fastmap will be destroyed
after attaching process.
Takahiro Kuwano [Tue, 7 Nov 2023 09:23:57 +0000 (18:23 +0900)]
mtd-utils: flash_erase: Add an option for JFFS2 cleanmarker size
JFFS2 supports buffer mode for ECC'd NOR Flash that cleanmarker size
is rounded up to mtd->writesize, while the '-j' option in flash_erase
utility uses fixed 12 bytes. That makes flash program ops unaligned to
mtd->writesize and causes program error or disables ECC.
The mkfs.jffs2 utility supports '-c' option that allows users to specify
cleanmarker size. This patch implements '-c' option in the flash_erase
that can be used along with '-j' option.
Signed-off-by: Takahiro Kuwano <Takahiro.Kuwano@infineon.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Piotr Esden-Tempski [Fri, 1 Sep 2023 00:55:03 +0000 (17:55 -0700)]
flashcp: Add write last option.
This option is useful for power fail resilient bootloader updates, for
systems that check only the partition header for validity before
attempting to load the bootloader. For example zynq.
Postponing the write of a specified amount of bytes at the beginning of
a bootloader makes sure a written bootloader slot is not considered
valid until all the data is written.
Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
projects / mtd-utils.git / log