Zhihao Cheng [Sun, 26 Jan 2025 06:42:01 +0000 (14:42 +0800)]
tests: checkfs: Add previous prototype for do_pwr_dn()
The compiler complains following message:
tests/checkfs/comm.c:37:5: warning: no previous prototype for
‘do_pwr_dn’
The function do_pwr_dn() is only used in tests/checkfs/checkfs.c, but
we still keep it in tests/checkfs/comm.c in case the function() is
used by more callers. Fix it by adding a previous declaration.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
The compiler compains following message:
jffsX-utils/jffs2reader.c:697:6: warning: potential null pointer
dereference
It won't bring any problems because the 'path' is guaranteed to be a
non-NULL variable. Fix the warning by adding a NULL pointer check.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
The compiler compains following message:
misc-utils/flashcp.c:439:3: warning: ‘wrlast_buf’ may be used
uninitialized in this function
It won't bring any problems because variable '‘wrlast_buf’ is always
initialized before being used. Fix the warning by setting 'wrlast_buf'
as NULL when declaring it.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Sun, 26 Jan 2025 06:41:58 +0000 (14:41 +0800)]
nand-utils: nanddump: Fix unexpected type compiling warning for argument 'dumpfile'
The const pointer argument cannot be passed into libc function 'free()',
otherwise the compiler will complain following message:
nand-utils/nanddump.c:168:10: warning: passing argument 1 of ‘free’
discards ‘const’ qualifier from pointer target type
Fixes: c89009463888 ("nanddump: don't leak copied command line arguments") Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Sun, 26 Jan 2025 06:41:57 +0000 (14:41 +0800)]
ubi-utils: mtdinfo: Fix type truncation compiling warning for 'start' variable
Fix following compiling warning by using the right format:
ubi-utils/mtdinfo.c:207:21: warning: format ‘%lx’ expects argument of
type ‘long unsigned int’, but argument 4 has type ‘long long unsigned
int’
Fixes: 78f4f38bd3de8 ("ubi-utils: Fix integer overflow in mtdinfo.c") Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Anton Moryakov [Tue, 24 Dec 2024 19:40:13 +0000 (22:40 +0300)]
ubi-utils: FIX DIVISION BY ZERO in ubinize.c
Report of the static analyzer:
Variable vi->alignment, whose possible value set allows a zero value at ubinize.c:375, is used as a denominator at ubinize.c:410.
If you look at the code more closely, it will be clear that the vi->alignment parameter is obtained from an external file passed as a command line argument.
A check was also performed if you pass a test.ini file of the following type to the input:
Corrections explained:
Updated the validation logic for vi->alignment:
- Replaced the check for negative alignment (`vi->id < 0`) with a more comprehensive check for non-positive alignment (`vi->alignment <= 0`).
- Updated the corresponding error message to reflect the requirement for a positive volume alignment.
This ensures more robust validation and improves error clarity when invalid alignment values are encountered.
Triggers found by static analyzer Svace.
Signed-off-by: Anton Moryakov <ant.v.moryakov@gmail.com> Reviewed-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Anton Moryakov [Tue, 24 Dec 2024 12:27:09 +0000 (15:27 +0300)]
jffsX-utils: fix integer overflow in jffs2dump.c
Report of the static analyzer:
The value of an arithmetic expression 'datsize + oobsize' is a subject to overflow because its operands are not cast to a larger data type before performing arithmetic.
Corrections explained:
- Added a check to validate datsize and oobsize to ensure they are non-negative and within a safe range.
- Cast datsize and oobsize to long before performing arithmetic to prevent potential integer overflow.
This change ensures safe computation of offsets and prevents undefined behavior caused by overflow.
Signed-off-by: Anton Moryakov <ant.v.moryakov@gmail.com> Reviewed-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Anton Moryakov [Sat, 14 Dec 2024 12:31:05 +0000 (15:31 +0300)]
ubi-utils: Fix integer overflow in mtdinfo.c
Report of the static analyzer:
The value of an arithmetic expression 'reginfo->offset + i * reginfo->erasesize' is a subject to overflow
because its operands are not cast to a larger data type before performing arithmetic
Corrections explained:
Added casting i and start to unsigned long long
Triggers found by static analyzer Svace.
Signed-off-by: Anton Moryakov <ant.v.moryakov@gmail.com> Reviewed-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Anton Moryakov [Sat, 14 Dec 2024 12:18:35 +0000 (15:18 +0300)]
nand-utils: Fix integer overflow in nandflipbits.c
Report of the static analyzer:
The value of an arithmetic expression 'bit_to_flip->block * mtd.eb_size + blkoffs' is a subject to overflow because its operands are not cast to a larger data type before performing arith$
Corrections explained:
Prevent arithmetic overflow in OOB read operation
Resolved an issue where the calculation of the offset in the OOB read operation could overflow due to operands not being cast to a larger data type. Specifically, the multiplication of bi$
Triggers found by static analyzer Svace.
Signed-off-by: Anton Moryakov <ant.v.moryakov@gmail.com> Reviewed-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Anton Moryakov [Wed, 11 Dec 2024 14:04:03 +0000 (17:04 +0300)]
misc-utils: flash_erase: FIX integer overflow in flash_erase.c
Report of the static analyzer:
The value of an arithmetic expression 'eb_cnt * mtd.eb_size' is a subject to overflow because its operands are not cast to a larger data type before performing arithmetic
Corrections explained:
Added explicit casting of eb_cnt to long long in the condition
if (eb_start == 0 && mtd.size == eb_cnt * mtd.eb_size)
to ensure the multiplication is performed in a 64-bit context,
preventing potential overflow for large values of eb_cnt and mtd.eb_size.
This ensures correct handling of devices with large block counts or block sizes.
Triggers found by static analyzer Svace.
Signed-off-by: Anton Moryakov <ant.v.moryakov@gmail.com> Reviewed-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Anton Moryakov [Tue, 10 Dec 2024 00:07:26 +0000 (03:07 +0300)]
ubi-utils: ubirsvol: Fix integer overflow in ubirsvol.c
Report of the static analyzer:
The value of an arithmetic expression 'vol_info.leb_size * args.lebs' is a subject to overflow because its operands are not cast to a larger data type before performing arithmetic
Corrections explained:
The fix ensures values are checked before multiplication.
Added casting vol_info.leb_size to long long
Triggers found by static analyzer Svace.
Signed-off-by: Anton Moryakov <ant.v.moryakov@gmail.com> Reviewed-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:30 +0000 (17:08 +0800)]
ubifs-utils: Support Address-Sanitizier debug
Add new option '--asan' for configuration to support dynamic
Address-Sanitizier debugging, which could detect kinds of invalid
memory accessing problems(eg. UAF, r/w OOB, etc.). Currently, only
ubifs-utils(mkfs.ubifs/fsck.ubifs) is supported.
Enable Address-Sanitizier debugging with configuration:
./configure --enable-asan
Notice: The Address-Sanitizier will stop the program and print
problems if memory problems are detected. Sometimes the memory
problems come from third libs(not mtd-utils), which could stuck
the testcases.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:27 +0000 (17:08 +0800)]
tests: ubifs_tools: mkfs_tests: Add fs content check test
Initialize UBIFS image from a given directory, then check whether the
fs content in mounted UBIFS is consistent with the original directory.
Both UBI volume and file are chosen as storage mediums to test.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:26 +0000 (17:08 +0800)]
tests: ubifs_tools: fsck_tests: Add bad images fixing test
For kinds of inconsistent UBIFS images(which can simulate corruptions
caused by some potentional UBIFS bug), check the result of fsck.
This testcase mainly checks whether the behavior is in expected after
repairing specific inconsistent UBIFS image.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
This is a preparation for adding bad images fsck testcase. There is
no debugfs tools (for example: debugfs[ext4], xfs_db) for UBIFS, so
there is no way to inject precise corruption into UBIFS image, we have
to prepare inconsistent UBIFS images in advance like e2fsprogs[1] does.
(Goto [2] to see how to generate inconsistent UBIFS images).
Original UBIFS image content:
/
├── corrupt_file (xattr - user.corrupt:123, 2K data)
├── dir
│ ├── block_dev
│ ├── char_dev
│ ├── dir
│ └── file (content: '123')
├── hardl_corrupt_file => corrupt_file
└── softl_corrupt_file -> corrupt_file
Here's a descriptons of the various testing images:
=========================================================================
image | Description | expectancy
-------------------------------------------------------------------------
good | good image contains | fsck success, fs content is
| kinds of files. | not changed.
-------------------------------------------------------------------------
sb_fanout | invalid fanout in | fsck failed.
| superblock. |
-------------------------------------------------------------------------
sb_fmt_version | invalid fmt_version | fsck failed.
| in superblock. |
-------------------------------------------------------------------------
sb_leb_size | invalid leb_size in | fsck failed.
| superblock. |
-------------------------------------------------------------------------
sb_log_lebs | invalid log lebs in | fsck failed.
| superblock. |
-------------------------------------------------------------------------
sb_min_io_size | invalid min_io_size | fsck failed.
| in superblock. |
-------------------------------------------------------------------------
master_highest_inum | invalid highest_inum| fsck success, fs content is
| in master nodes. | not changed.
-------------------------------------------------------------------------
master_lpt | bad lpt pos in | fsck success, fs content is
| master nodes. | not changed.
-------------------------------------------------------------------------
master_tnc | bad tnc pos in | fsck success, fs content is
| master nodes. | not changed.
-------------------------------------------------------------------------
master_total_dead | bad total_dead in | fsck success, fs content is
| master nodes. | not changed.
-------------------------------------------------------------------------
master_total_dirty | bad total_dirty in | fsck success, fs content is
| master nodes. | not changed.
-------------------------------------------------------------------------
master_total_free | bad total_free in | fsck success, fs content is
| master nodes. | not changed.
-------------------------------------------------------------------------
journal_log | corrupted log area. | fsck success, fs content is
| | not changed.
-------------------------------------------------------------------------
journal_bud | corrupted bud area. | fsck success, file data is
| | lost.
-------------------------------------------------------------------------
orphan_node | bad orphan node. | fsck success, file is
| | deleted as expected.
-------------------------------------------------------------------------
lpt_dirty | bad dirty in pnode. | fsck success, fs content is
| | not changed.
-------------------------------------------------------------------------
lpt_flags | bad flags in pnode | fsck success, fs content is
| (eg. index). | not changed.
-------------------------------------------------------------------------
lpt_free | bad free in pnode. | fsck success, fs content is
| | not changed.
-------------------------------------------------------------------------
lpt_pos | bad pos in nnode. | fsck success, fs content is
| | not changed.
-------------------------------------------------------------------------
ltab_dirty | bad dirty in lprops | fsck success, fs content is
| table. | not changed.
-------------------------------------------------------------------------
ltab_free | bad free in lprops | fsck success, fs content is
| table. | not changed.
-------------------------------------------------------------------------
index_size | bad index size in | fsck success, fs content is
| master nodes. | not changed.
-------------------------------------------------------------------------
tnc_lv0_key | bad key in lv0 | fsck success, fs content is
| znode. | not changed.
-------------------------------------------------------------------------
tnc_lv0_len | bad len in lv0 | fsck success, fs content is
| znode. | not changed.
-------------------------------------------------------------------------
tnc_lv0_pos | bad pos in lv0 | fsck success, fs content is
| znode. | not changed.
-------------------------------------------------------------------------
tnc_noleaf_key | bad key in non-leaf | fsck success, fs content is
| znode. | not changed.
-------------------------------------------------------------------------
tnc_noleaf_len | bad len in non-leaf | fsck success, fs content is
| znode. | not changed.
-------------------------------------------------------------------------
tnc_noleaf_pos | bad pos in non-leaf | fsck success, fs content is
| znode. | not changed.
-------------------------------------------------------------------------
corrupted_data_leb | corrupted data leb. | fsck success, partial data of
| | file is lost.
-------------------------------------------------------------------------
corrupted_idx_leb | corrupted index leb.| fsck success, fs content is
| | not changed.
-------------------------------------------------------------------------
inode_data | bad data node. | fsck success, file content
| | is changed, other files are
| | not changed.
-------------------------------------------------------------------------
inode_mode | bad inode mode for | fsck success, file is
| file. | dropped, other files are not
| | changed.
-------------------------------------------------------------------------
inode_nlink | wrong nlink for | fsck success, nlink is
| file. | corrected, fs content is not
| | changed.
-------------------------------------------------------------------------
inode_size | wrong inode size | fsck success, inode size is
| for file. | corrected, fs content is not
| | changed.
-------------------------------------------------------------------------
inode_xcnt | wrong inode | fsck success, xattr_cnt is
| xattr_cnt for file. | corrected, fs content is not
| | changed.
-------------------------------------------------------------------------
soft_link_inode_mode| bad inode mode for | fsck success, soft link
| solf link file. | file is dropped, other files
| | are not changed.
-------------------------------------------------------------------------
soft_link_data_len | bad inode data_len | fsck success, soft link
| for solt link file. | file is dropped, other files
| | are not changed.
-------------------------------------------------------------------------
dentry_key | bad dentry key for | fsck success, dentry is
| file. | removed, other files are
| | not changed.
-------------------------------------------------------------------------
dentry_nlen | inconsistent nlen | fsck success, dentry is
| and name in dentry | removed, other files are
| for file. | not changed.
-------------------------------------------------------------------------
dentry_type | inconsistent type | fsck success, dentry is
| between dentry and | removed, other files are
| inode for file. | not changed.
-------------------------------------------------------------------------
xinode_flags | lost UBIFS_XATTR_FL | fsck success, xattr is
| in xattr inode | removed, other files are
| flags for file. | not changed.
-------------------------------------------------------------------------
xinode_key | bad xattr inode key | fsck success, xattr is
| for file. | removed, other files are
| | not changed.
-------------------------------------------------------------------------
xinode_mode | bad xattr inode | fsck success, xattr is
| mode for file. | removed, other files are
| | not changed.
-------------------------------------------------------------------------
xentry_key | bad xattr entry key | fsck success, xattr is
| for file. | removed, other files are
| | not changed.
-------------------------------------------------------------------------
xentry_nlen | inconsistent nlen | fsck success, xattr is
| and name in xattr | removed, other files are
| entry for file. | not changed.
-------------------------------------------------------------------------
xentry_type | inconsistent type | fsck success, xattr is
| between xattr entry | removed, other files are
| and xattr inode for | not changed.
| file. |
-------------------------------------------------------------------------
xent_host | the xattr's host | fsck success, file, hard
| is a xattr too, the | link and soft link are
| flag of corrupt_file| dropped, other files are
| inode is modified. | not changed.
-------------------------------------------------------------------------
dir_many_dentry | dir has too many | fsck success, hard link is
| dentries, the dentry| dropped, other files are not
| of hard link is | changed.
| modified. |
-------------------------------------------------------------------------
dir_lost | bad dentry for dir. | fsck success, the 'file' is
| | recovered under lost+found,
| | left files under dir are
| | removed, other files are not
| | changed.
-------------------------------------------------------------------------
dir_lost_duplicated | bad inode for dir, | fsck success, the 'file' is
| there is a file | recovered with INO_<inum>_1
| under lost+found, | under lost+found, left files
| which named with the| under dir are removed, other
| inum of the 'file'. | files are not changed.
-------------------------------------------------------------------------
dir_lost_not_recover| bad inode for dir, | fsck success, all files
| lost+found is a | under dir are removed,
| regular file and | other files are not changed.
| exists under root |
| dir. |
-------------------------------------------------------------------------
root_dir | bad '/'. | fsck success, create new
| | root dir('/'). All regular
| | files are reocovered under
| | lost+found, other files are
| | removed.
-------------------------------------------------------------------------
empty_tnc | all files have bad | fsck success, fs content
| inode. | becomes empty.
=========================================================================
Zhihao Cheng [Mon, 11 Nov 2024 09:08:24 +0000 (17:08 +0800)]
tests: ubifs_tools: fsck_tests: Add random_corrupt+fsck test
Inject random corruption on UBIFS image by writting random data on
kinds of mtd devices (eg. nand, nor), check the consistency of UBIFS
after fsck.
This testcase simulates random bad UBIFS image caused by hardware
exceptions(eg. ecc uncorrectable, unwritten), and makes sure that
fsck.ubifs could make UBIFS be consistent after repairing UBIFS
image.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:23 +0000 (17:08 +0800)]
tests: ubifs_tools: fsck_tests: Add cycle_powercut+fsck test
Inject powercut while doing fsstress on mounted UBIFS, check the
consistency of UBIFS after fsck.
This testscase mainly makes sure that fsck.ubifs can make UBIFS
image be consistent in common stress cases and powercut cases.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:22 +0000 (17:08 +0800)]
tests: ubifs_tools: fsck_tests: Add corrupt+fsck+fault_inject test
Inject memory/io fault while doing fsck for corrupted UBIFS images.
This testcase mainly checks whether fsck.ubifs has problems (eg.
UAF, null-ptr-def, etc.) in random error paths. Besides, it provides
a similar way to simulate powercut during fsck, and checks whether
the fsck.ubifs can fix an UBIFS image after many rounds interrupted
by kinds of errors.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:21 +0000 (17:08 +0800)]
tests: ubifs_tools: fsck_tests: Add powercut+fsck+mount test
Inject powercut while doing fsstress on mounted UBIFS for kinds of
flashes (eg. nand, nor).
This testcase mainly makes sure that fsck.ubifs can make UBIFS image
be consistent on different flashes (eg. nand, nor). Because the
min_io_size of nor flash is 1, the UBIFS image on nor flash will be
different from nand flash after doing powercut, so we need make sure
fsck.ubifs can handle these two types of flash.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:20 +0000 (17:08 +0800)]
tests: ubifs_tools: fsck_tests: Add cycle mount+fsck test
Do fsstress and fsck, check whether there are any files(and their data)
are lost after fsck. This testcase mainly checks whether fsck.ubifs could
corrupt the filesystem content in common case.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:18 +0000 (17:08 +0800)]
tests: Add common libs for testing fsck.ubifs/mkfs.ubifs
This is a preparation for adding testcases for fsck.ubifs and
mkfs.ubifs. Add some common functions, for example: powercut,
load_mtdram, mount_ubifs, encryption operations, etc.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:16 +0000 (17:08 +0800)]
fsck.ubifs: Do final committing
This is the 18/18 step of fsck. Do final committing, commit problem
fixing modifications(which are generated since step 14) to disk, and
clear %UBIFS_MST_DIRTY flag for master node.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:15 +0000 (17:08 +0800)]
fsck.ubifs: Handle disconnected files
This is the 17/18 step of fsck. Recover disconnected files into
lost+found. If there is no free space left to recover the disconnected
files, fsck may delete the files to make filesystem be consistent.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Huang Xiaojia [Mon, 11 Nov 2024 09:08:14 +0000 (17:08 +0800)]
fsck.ubifs: Check and create the lost+found
This is the 16/18 step of fsck. Check whether the lost+found is existed,
create a new one if it is not found. This step makes sure that disconnected
file can be recovered under the lost+found.
Zhihao Cheng [Mon, 11 Nov 2024 09:08:13 +0000 (17:08 +0800)]
fsck.ubifs: Check and create the root dir
This is the 15/18 step of fsck. Check whether the root dir is existed,
create a new one if it is not found. This step makes sure that filesystem
can be mounted successful.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:12 +0000 (17:08 +0800)]
ubifs-utils: libubifs: Support some file operations
Add some file operations, such as ubifs_lookup, ubifs_mkdir, etc., this
is a preparation for recovering disconnected files or root dir in fsck.
File writing operations are based on the journal subsystem, generated
dirty data depends on a new commit in subsequent steps to update disk
content.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:11 +0000 (17:08 +0800)]
fsck.ubifs: Check and correct the index size
This is the 14/18 step of fsck. Check and correct the index size by
traversing TNC just like dbg_check_idx_size does. This step should
be executed after first committing, because 'c->calc_idx_sz' can be
changed in 'ubifs_tnc_start_commit' and the initial value of
'c->calc_idx_sz' read from disk is untrusted.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:09 +0000 (17:08 +0800)]
fsck.ubifs: check and correct the space statistics
This is the 12/18 step of fsck. Check and correct the space statistics.
There could be following steps and possible errors:
Step 1. Exit for check mode, if %FR_LPT_CORRUPTED or %FR_LPT_INCORRECT
is set in lpt status, the exit code should have %FSCK_UNCORRECTED.
Step 2. Check lpt status, if %FR_LPT_CORRUPTED is set in lpt status,
normal mode with 'no' answer will exit, other modes will rebuild lpt.
Step 3. Traverse LPT nodes, check the correctness of nnode and pnode,
compare LEB scanning result with LEB properties.
a. LPT node is corrupted, normal mode with 'no' answer will exit,
rebuild lpt for other modes.
b. Incorrect nnode/pnode, normal mode with 'no' answer will exit,
other other modes will correct the nnode/pnode.
c. Inconsistent comparing result, normal mode with 'no' answer
will exit, other modes will correct the space statistics.
Step 4. Check and correct the lprops table information.
Step 5. Set gc lnum(ubifs_rcvry_gc_commit / take_gc_lnum).
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:08 +0000 (17:08 +0800)]
fsck.ubifs: Move common functions and data structures into check_space.c
This is a preparation for adding LPT checking support. Move some data
structures and functions into check_space.c, also factor out some common
functions in libubifs:
1. Move 'lpts' from rebuild module, make it resuable for non-rebuild_fs
modes.
2. Move function 'get_free_leb' from rebuild_fs.c, it could be reused in
building LPT.
3. Move function 'build_lpt' from rebuild_fs.c, it could be reused in
building LPT.
4. Factor out lpt nodes freeing into a new function ubifs_free_lpt_nodes.
5. Factor out nnode dirty marking implementations into a new function
ubifs_make_nnode_dirty.
5. Export the function of nnode number calculation, calc_nnode_num is
renamed as ubifs_calc_nnode_num.
6. Export the function of making pnode dirty, do_make_pnode_dirty is
renamed as ubifs_make_pnode_dirty.
7. Rename next_pnode_to_dirty to ubifs_find_next_pnode and export it.
8. Export free_buds and expend its parameters.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:07 +0000 (17:08 +0800)]
fsck.ubifs: Check whether the TNC is empty
This is the 11/18 step of fsck. Check whether the TNC is empty, turn to
rebuild_fs if it is not found. Can we recreate a new root dir to avoid
empty TNC? The answer is no, lpt fixing should be done before creating
new entry, but lpt fixing needs a committing before new dirty data
generated to ensure that bud data won't be overwritten(bud LEB could
become freeable after replaying journal, corrected lpt may treat it as
a free one to hold new data, see details in space checking & correcting
step). Then we have to create the new root dir after fixing lpt and a
committing, znode without children(empty TNC) maybe written on disk at
the moment of committing, which corrupts the UBIFS image. So we choose
to rebuild the filesystem if the TNC is empty, this case is equivalent
to corrupted TNC.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Xiang Yang [Mon, 11 Nov 2024 09:08:06 +0000 (17:08 +0800)]
fsck.ubifs: Check and correct files' information
This is the 10/18 step of fsck. Check and handle inconsistent files, the
checking rule is same as rebuild mode which has been implemented in
check_and_correct_files, but the methods of handling are different:
1. Correct the file information for safe mode, danger mode and normal
mode with 'yes' answer, other modes will exit.
Signed-off-by: Xiang Yang <xiangyang3@huawei.com> Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:05 +0000 (17:08 +0800)]
fsck.ubifs: Check and handle unreachable files
This is the 9/18 step of fsck. Check and handle unreachable files, the
checking rule is same as rebuild mode which has been implemented in
file_is_reachable, but the methods of handling are different:
1. Move unreachable regular file into disconnected list, let subsequent
steps to handle them with lost+found.
2. Delete unreachable non-regular file.
3. Delete unreachable directory entries.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:04 +0000 (17:08 +0800)]
fsck.ubifs: Check and handle invalid files
This is the 8/18 step of fsck. Check and handle invalid files, the
checking rule is same as rebuild mode which has been implemented in
file_is_valid, but the methods of handling are different:
1. Move unattached(file has no dentries) regular file into disconnected
list, let subsequent steps to handle them with lost+found.
2. Make file type be consistent between inode, detries and data nodes by
deleting dentries or data blocks.
3. Delete file for other invalid cases(eg. file has no inode).
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:02 +0000 (17:08 +0800)]
fsck.ubifs: Ensure that TNC LEB can be scanned successful
This is the second part of 6/18 step in fsck. Add an extra checking for
non-check mode while traversing TNC, make sure that all LEBs(contains TNC)
can be scanned successful. There could be following steps and possible
errors:
Step 2. Scan all LEBs(contain TNC), remove TNC branch which points to
corrupted LEB.
a. corrupted node is found by scanning: If current node is index node,
danger mode with rebuild_fs and normal mode with 'yes' answer will
turn to rebuild filesystem, other modes will exit; If current node
is non-index node, danger mode and normal mode with 'yes' answer
will remove all TNC branches which point to the corrupted LEB,
other modes will exit.
b. LEB contains both index and non-index nodes: danger mode with
rebuild_fs and normal mode with 'yes' answer will turn to rebuild
filesystem, other modes will exit.
This is a preparation for space checking, which means that ubifs_scan
will always succeed when check properties for any TNC LEBs. We do this
before checking files(step 7) & extracting dentry tree(step 8), nodes
cannot be dropped(which may corrupted file and make file inconsistent
again) when scanning corrupted as long as the dentry tree is extracted.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:01 +0000 (17:08 +0800)]
fsck.ubifs: Traverse TNC and construct files
This is the 6/18 step of fsck. Traverse TNC and construct files. There
could be following steps and possible errors:
Step 1. Traverse TNC, check whether the leaf node is valid, remove invalid
nodes, construct file for valid node and insert file into file tree.
a. corrupted node searched from TNC: remove corresponding TNC branch for
danger mode and normal mode with 'yes' answer, other modes will exit.
b. corrupted index node read from TNC: danger mode with rebuild_fs and
normal mode with 'yes' answer will turn to rebuild filesystem, other
modes will exit.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:08:00 +0000 (17:08 +0800)]
fsck.ubifs: Move common functions and data structures into fsck.ubifs.c
This is a preparation for adding TNC checking support. Following data
structures and functions are moved into fsck.ubifs.c:
1. Move 'scanned_files' and 'used_lebs' from rebuild module, make them
resuable for non-rebuild_fs modes.
2. Move function 'handle_error' from load_fs.c, it could be reused in
other steps.
3. Add new function ubifs_tnc_remove_node in libubifs, which could
remove index entry for a node by given position.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:07:59 +0000 (17:07 +0800)]
fsck.ubifs: Recover isize
This is the 5/18 step of fsck. Recover isize. There could be following
steps and possible errors:
Step 1. Traverse size tree, lookup corresponding inode from TNC
a. corrupted node searched from TNC: skip node for danger mode and
normal mode with 'yes' answer, other modes will exit.
b. corrupted index node read from TNC: danger mode with rebuild_fs and
normal mode with 'yes' answer will turn to rebuild filesystem, other
modes will exit.
Step 2. update isize for inode. Keep <inum, isize> in size tree for check
mode, update inode node in place for other modes.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:07:58 +0000 (17:07 +0800)]
fsck.ubifs: Consolidate log
This is the 4/18 step of fsck. Consolidate log to ensure enough space
in log area. There could be following possible errors:
1. corrupted scanning data in log area: danger mode with rebuild_fs and
normal mode with 'yes' answer will turn to rebuild filesystem, other
modes will exit.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:23 +0000 (17:01 +0800)]
fsck.ubifs: Handle orphan nodes
This is the 3/18 step of fsck. Handle orphan nodes, update TNC & LPT.
There could be following steps and possible errors:
Step 1. scan orphan LEB, get all orphan nodes
a. corrupted scanning data in orphan area: danger mode and normal mode
with 'yes' answer will drop orphan LEB, other modes will exit.
Step 2. parse orphan node, find the original inode for each inum
a. corrupted node searched from TNC: skip node for danger mode and
normal mode with 'yes' answer, other modes will exit.
b. corrupted index node read from TNC: danger mode with rebuild_fs and
normal mode with 'yes' answer will turn to rebuild filesystem, other
modes will exit.
Step 4. remove inode for each inum, update TNC & LPT
a. corrupted index node read from TNC: danger mode with rebuild_fs and
normal mode with 'yes' answer will turn to rebuild filesystem, other
modes will exit.
b. corrupted lpt: Set %FR_LPT_CORRUPTED for lpt status. Ignore the
error.
c. incorrect lpt: Set %FR_LPT_INCORRECT for lpt status. Ignore the
error.
d. If lpt status is not empty, skip updating lpt.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:22 +0000 (17:01 +0800)]
fsck.ubifs: Replay journal
This is the 2/18 step of fsck. Replay journal, update TNC & LPT.
There could be following steps and possible errors:
Step 1. scan log LEB, get all bud LEBs
a. corrupted scanning data in log area: danger mode with rebuild_fs and
normal mode with 'yes' answer will turn to rebuild filesystem, other
modes will exit.
Step 2. scan bud LEBs, get all nodes
a. corrupted scanning data in bud LEB: danger mode and normal mode with
'yes' answer will drop bud LEB and set %FR_LPT_INCORRECT for lpt
status, other modes will exit.
Step 3. apply nodes, record latest isize into size_tree
Step 4. apply nodes, update TNC & LPT
a. corrupted data searched from TNC: skip node and set %FR_LPT_INCORRECT
lpt status for danger mode and normal mode with 'yes' answer, other
modes will exit.
b. corrupted index node read from TNC: danger mode with rebuild_fs and
normal mode with 'yes' answer will turn to rebuild filesystem, other
modes will exit.
c. corrupted lpt: Set %FR_LPT_CORRUPTED for lpt status. Ignore the
error.
d. incorrect lpt: Set %FR_LPT_INCORRECT for lpt status. Ignore the
error.
e. If lpt status is not empty, skip updating lpt.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:21 +0000 (17:01 +0800)]
fsck.ubifs: Read master node & init lpt
This is the 1/18 step of fsck. Read and check master node, init lpt.
There could be following errors:
1. corrupted scanning data in master area or invalid master node:
danger mode with rebuild_fs and normal mode with 'yes' answer will
turn to rebuild filesystem, other modes will exit.
2. incorrect space statistics in master node: Set %FR_LPT_INCORRECT for
for lpt status. Ignore the error.
3. corrupted lpt: Set %FR_LPT_CORRUPTED for lpt status. Ignore the error.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:20 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Write master node
This is the 12/12 step of rebuilding. Since all meta areas are ready,
master node can be updated. After this step, a consistent UBIFS image
can be mounted, and it should pass all tests from chk_fs, chk_general,
chk_index, chk_lprops and chk_orphans.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:19 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Clean up log and orphan area
This is the 11/12 step of rebuilding. Clean up log and orphan area, all
nodes have been recovered, these two areas should be cleared, otherwise
old content in journal/orphan could be replayed in next mounting.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:18 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Build LPT
This is the 10/12 step of rebuilding. All LEBs' properties can be
calculated in previous steps according to all nodes' position, then
construct LPT just like mkfs does, and write LPT on flash.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:17 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Build TNC
This is the 9/12 step of repairing. Construct TNC according to scanned
files, and write TNC on flash, just like mkfs does.
Building TNC can effectively solve many failed mounting problems caused
by bad TNC (eg. bad node pointed by TNC, bad key order in znode, etc.).
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:15 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Re-write data
This is the 8/12 step of rebuilding. Re-write data. Read data from
LEB and write back data, make sure that all LEB is ended with empty
data(0xFF). It will prevent failed gc scanning in next mounting.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:14 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Record used LEBs
This is the 7/12 step of rebuilding. Record used LEBs which may hold
useful nodes, then left unused LEBs could be taken for storing new index
tree. Notice, LEB that contains effective nodes on deleted trees in step
1 is regarded as used.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:13 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Check and correct files' information
This is the 6/12 step of rebuilding. Correct the file information.
Traverse all files and calculate information (nlink, size, xattr_cnt,
etc.) for each file just like check_leaf() does, correct inode node
based on the calculated information.
Now, all files are consistent, and UBIFS will pass chk_fs after mounting.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:12 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Extract reachable directory entries tree
This is the 5/12 step of rebuilding. Extract reachable directory entries
tree. Make sure that all files can be searched from '/', unreachable
file is deleted. So, all files can be accessible in userspace after
reparing.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:11 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Filter invalid files
This is the 4/12 step of rebuilding. Filter out invalid files and drop
them, for example:
1. File has no inode node or inode nlink is zero
2. Nonconsistent file type between inode node and dentry nodes
3. File has no dentry nodes(excepts '/')
4. Encrypted file has no xattr information
5. Non regular file has data nodes
6. Directory/xattr file has more than one dentries
7. Xattr file has no host inode, or the host inode is a xattr
...
Valid xattr file will be inserted into corresponding host file's subtree.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:10 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Add valid nodes into file
This is the 3/12 step of rebuilding. Generate file according to left valid
inode nodes and dentry nodes. Based on the results from step 2, it is easy
to understand:
Step 2 has done:
valid_inos - del_inos = left_inos
valid_dents - del_dents = left_dents
Step 3 should do:
Traverse left_inos and left_dents, insert inode/dentry nodes into
corresponding file.
After that, all files are generated by scanning, the next thing to do is
dropping invalid files(eg. nonconsistent file type between inode node and
dentry nodes, file has no dentry nodes(excepts '/'), encrypted file has
no xattr information, etc.), which will be done in next step.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:09 +0000 (17:01 +0800)]
fsck.ubifs: rebuild_fs: Remove deleted nodes from valid node tree
This is the 2/12 step of rebuilding. Traverse nodes from del_inos and
del_dents trees, remove inode nodes and dentry nodes with smaller sqnum
from valid trees.
This step handles deleting case, for example, file A is deleted, deleted
inode node and deleted dentry node are written, if we ignore the deleted
nodes, file A can be recovered after rebuilding because undeleted inode
node and undeleted dentry node can be scanned. There's an exception, if
deleted inode node and deleted dentry node are reclaimed(by gc) after
deletion, file A is recovered. UBIFS rebuild_fs cannot solve it, because
the real existence information of nodes depends on TNC, but TNC should
not be depended for UBIFS rebuild_fs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:08 +0000 (17:01 +0800)]
fsck.ubifs: Add rebuilding filesystem support
Add rebuilding filesystem support. This is the 1/12 step of rebuilding.
Collect files, valid inode nodes, deleted inode nodes, valid dentry
nodes and deleted dentry nodes in kinds of trees by scanning nodes from
flash. Corrupted nodes(eg. incorrect crc, bad inode size, bad dentry
name length, etc.) are dropped during scanning. Larger sqnum node is
picked when more than 1 nodes with same index.
In this step, trun node and data nodes are put into corresponding file,
inode/dentry nodes are put into four trees: valid_inos(nlink != 0),
del_inos(nlink is 0), valid_dents(inum != 0), del_dents(inum is 0).
Next step will process above four trees to deal deletion situations.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:07 +0000 (17:01 +0800)]
fsck.ubifs: Add file organization realization
In order to check the consistency of each file and the reachability of
the whole dentry tree, fsck orginizes all nodes into files. And the
final recovered file(xattr is treated as a file) is organized as:
(rbtree, inum indexed)
/ \
file1 file2
/ \
file3 file4
file {
inode node // each file has 1 inode node
dentry (sub rb_tree, sqnum indexed) // '/' has no dentries,
// otherwise at least 1
// dentry is required.
trun node // the newest one truncation node
data (sub rb_tree, block number indexed) // Each file may have 0
// or many data nodes
xattrs (sub rb_tree, inum indexed) // Each file may have 0 or
// many xattr files
}
Each file from file rb_tree is constructed by scanning nodes(eg. inode,
dentry, etc.) from the TNC or the UBI volume. File's xattrs will be
initialized in subsequent steps.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:06 +0000 (17:01 +0800)]
fsck.ubifs: Add node parsing functions
Add parsing functions for each type of nodes, which will be used for
checking the validity of raw node data while reading from TNC or
scanning from UBIFS logical erase block.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:05 +0000 (17:01 +0800)]
fsck.ubifs: Load filesystem information from UBI volume
Load filesystem information from UBI volume (Similar to UBIFS mounting
process), initialize kinds of buffers and read superblock. This is the
base step for both fsck and rebuild_fs. Subsequent pacthes will complete
this step by adding more steps(eg. read master, replay journal, etc.)
which are only used in fsck.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:04 +0000 (17:01 +0800)]
fsck.ubifs: Distinguish reasons when certain failures happen
Read failure caused by scanning corrupted data or invalid data members
should be identified, because fsck can handle it. Updating lp failure
caused by bad space statistics should be identified too, because fsck
can handle it.
Add eight callback functions to implement it for fsck:
1. set_failure_reason_callback: Record failure reasons when reading or
parsing node failed, there are four reasons:
a. FR_DATA_CORRUPTED: scanning corrupted data or invalid nodes found
b. FR_TNC_CORRUPTED: invalid index nodes
c. FR_LPT_CORRUPTED: invalid pnode/nnode
d. FR_LPT_INCORRECT: invalid space statistics or invalid LEB properties
2. get_failure_reason_callback: get failure reasons
3. clear_failure_reason_callback: Clear the error which is caused by
above reasons.
4. test_and_clear_failure_reason_callback: Check and clear the error
which is caused by above reasons, if so, fsck will handle it
according to specific situation.
For example, fsck will drop data node rather than fails to return
when reading failure is caused by DATA_CORRUPTED.
For another example, journal replaying will continue rather than
fails to return if updating lpt failure is caused by LPT_CORRUPTED.
5. set_lpt_invalid_callback: Set the invalid lpt status
6. test_lpt_valid_callback: Check whether the lpt is corrupted/incorrect,
it should be invoked before updating lp, if lpt status is invalid,
returns false (which means that caller should skip updating lp, because
updating lp could trigger assertion failed in ubifs_change_lp).
7. can_ignore_failure_callback: Check whether the failure can be
ignored, some inconsistent errors won't affect the fsck process,
for example wrong space statistics can be fixed after traversing
TNC, so failures caused by incorrect space statistics can be ignored.
8. handle_failure_callback: Check whether the failure can be handled,
some inconsistent errors could be fixed by fsck, we have fix_problem
to do that, but UBIFS needs a callback function to invoke it in common
libs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:03 +0000 (17:01 +0800)]
fsck.ubifs: Add inconsistent problem handling asking function
There are four dimensions to define the type of inconsistent problems:
1. fixable: Some inconsistent problems can't be fixed, for example
corrupted superblock. Un-fixable problem will abort program.
2. must fix: Some inconsistent problems can be ignored(eg. incorrect
isize), but some are not(eg. corrupted TNC), which will affect the
subsequent fsck steps.
3. drop data: Some fixing methods will drop user data, which is
unacceptable for safe mode. If it happens, fsck will be aborted.
4. need rebuild: Some inconsistent problems depends on rebuilding
filesystem to be fixed(eg. corrupted master node, corrupted TNC).
Define an asking function to handle above kinds of inconsistent problems.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:02 +0000 (17:01 +0800)]
fsck.ubifs: Add fsck support
Add basic process code for fsck.ubifs. There are following modes for fsck:
1. normal mode: Check the filesystem, ask user whether or not to fix
the problem as long as inconsistent data is found during fs checking.
2. safe mode: Check and safely repair the filesystem, if there are any
data dropping operations needed by fixing, fsck will fail.
3. danger mode: Answer 'yes' to all questions. There two sub modes:
a) Check and repair the filesystem according to TNC, data dropping
will be reported. If TNC/master/log is corrupted, fsck will fail.
b) Check and forcedly repair the filesystem according to TNC, turns
to rebuild filesystem if TNC/master/log is corrupted. Always make
fsck succeed.
4. check mode: Make no changes to the filesystem, only check the
filesystem.
5. rebuild mode: Scan entire UBI volume to find all nodes, and rebuild
filesystem, always make fsck success.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:01:00 +0000 (17:01 +0800)]
ubifs-utils: Replace ubifs related source code with linux kernel implementation
Replace ubifs related source code with the implementation of linux kernel.
It makes userspace implementation be same with linux kernel, then
fsck.ubifs can resuse the code.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:00:57 +0000 (17:00 +0800)]
ubifs-utils: Adapt gc subsystem in libubifs
Adapt gc subsystem(find.c, gc.c, scan.c) in libubifs, compared with
linux kernel implementations:
1. Adapt print_hex_dump based on implementations in hexdump.c.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:00:56 +0000 (17:00 +0800)]
ubifs-utils: Adapt orphan.c in libubifs
Adapt orphan.c in libubifs, compared with linux kernel implementations:
1. Keep the commit related implementations, because do_commit depends
on these functions which will be invoked in fsck.
2. Keep the orphan replaying implementations, because fsck needs them.
3. Other implementations are removed which won't be used in mkfs/fsck.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:00:55 +0000 (17:00 +0800)]
ubifs-utils: Adapt misc.h in libubifs
Adapt misc.h in libubifs, compared with linux kernel implementations:
1. Remove some functions(eg. ubifs_compr_name, ubifs_wake_up_bgt) which
won't be used in fsck/mkfs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 09:00:54 +0000 (17:00 +0800)]
ubifs-utils: Adapt master.c in libubifs
Adapt master.c in libubifs, compared with linux kernel implementations:
1. Remove authentication related implementations
(eg. mst_node_check_hash), authentication is not supported in fsck
for now.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:10 +0000 (16:37 +0800)]
ubifs-utils: Adapt key.h in libubifs
Adapt key.h in libubifs, compared with linux kernel implementations:
1. Add '__unused' modifier for unused parameters to avoid compiling
warnings.
2. Remove some functions(eg. lowest_dent_key, dent_key_init_flash)
which won't be used in fsck/mkfs.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:09 +0000 (16:37 +0800)]
ubifs-utils: Adapt debug subsystem in libubifs
Adapt debug subsystem(debug.c, debug.h) in libubifs, compared with
linux kernel implementations:
1. Only keep the disk data and space statistics dumping implementations,
dbg_walk_index and add_size which are used by fsck, other debuging
related implementations and sysfs interfaces are removed, because
fsck will check fs in another way.
2. Change the behavior of ubifs_assert_failed(), make filesystem
readonly when assertion is failed.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:08 +0000 (16:37 +0800)]
ubifs-utils: Adapt commit.c in libubifs
Adapt commit.c in libubifs, compared with linux kernel implementations:
1. Remove debug related implementations(eg. dbg_check_old_index), debug
functions are not needed by fsck, because fsck will check fs in
another way.
2. Remove ubifs background committing related implementations, there
will be no background ubifs threads in fsck.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:07 +0000 (16:37 +0800)]
ubifs-utils: Adapt budget.c in libubifs
Adapt budget.c in libubifs, compared with linux kernel implementations:
1. Remove writeback related functions, there are no dirty pages/inodes
for UBIFS in userspace process.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:06 +0000 (16:37 +0800)]
ubifs-utils: Adapt journal.c in libubifs
Adapt journal.c in libubifs, compared with linux kernel implementations:
1. Remove all ubifs_jnl_XXX functions. Only keep the basic space
reservation code, fsck will add new functions for journaling
operations without using linux in-memory inode/dentry.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:05 +0000 (16:37 +0800)]
ubifs-utils: Adapt dir.c in libubifs
Adapt dir.c in libubifs, compared with linux kernel implementations:
1. Remove all functions. Only keep an empty source file, fsck will
add new functions for mkdir/link operations without using linux
in-memory inode/dentry.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:04 +0000 (16:37 +0800)]
ubifs-utils: Adapt auth.c in libubifs
Adapt auth.c in libubifs, compared with linux kernel implementations:
1. Only keep implementations used by mkfs, other implementations
are removed.
2. Adapt functions based on implementations in common/sign.c.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:03 +0000 (16:37 +0800)]
ubifs-utils: Adapt sb.c in libubifs
Adapt sb.c in libubifs, compared with linux kernel implementations:
1. Remove authentication related implementations
(eg. authenticate_sb_node), authentication is not supported in fsck
for now.
2. Remove some functions(eg. create_default_filesystem) which won't be
used in fsck/mkfs.
3. Modify ubifs_read_superblock(), remove some assignments which won't
be used in mkfs/fsck.
4. Apapt fixup_leb to ignore %-EBADMSG, subsequent steps will check
all areas carefully.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:02 +0000 (16:37 +0800)]
ubifs-utils: Adapt recovery subsystem in libubifs
Adapt recovery subsystem(replay.c, recovery.c) in libubifs, compared with
linux kernel implementations:
1. Remove authentication related implementations
(eg. authenticate_sleb_hash), authentication is not supported in fsck
for now.
2. Add explicit type conversions(const char *) to avoid compiling
warnings.
3. Replace implementations of inode_fix_size() with ubifs_assert(0),
authentication is not supported in fsck, so this function won't
be invoked.
4. Remove unused ubifs_clean_lebs() and ubifs_write_rcvrd_mst_node().
5. Adapt fix_unclean_leb/recover_head/fix_size_in_place to ignore
%-EBADMSG, subsequent steps will check nodes in lpt/main area
carefully.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:01 +0000 (16:37 +0800)]
ubifs-utils: Adapt log.c in libubifs
Adapt log.c in libubifs, compared with linux kernel implementations:
1. Remove debug related implementations(eg. dbg_check_bud_bytes), debug
functions are not needed by fsck, because fsck will check fs in
another way.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:37:00 +0000 (16:37 +0800)]
ubifs-utils: Adapt tnc subsystem in libubifs
Adapt tnc subsystem(tnc.c,tnc_misc.c,tnc_commit.c) in libubifs, compared
with linux kernel implementations:
1. Remove debug related functions(eg. dbg_check_inode_size), debug
functions are not needed by fsck, because fsck will check fs in
another way.
2. Remove some functions(eg. ubifs_tnc_bulk_read) which won't be used
in fsck/mkfs.
3. Adapt tnc_delete and ubifs_search_zbranch to handle empty TNC case,
which could happen in fsck.
4. Don't skip checking the length of non-leaf index node's branch in
read_znode.
5. Adapt try_read_node to ignore %-EBADMSG, subsequent steps will check
nodes carefully.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:59 +0000 (16:36 +0800)]
ubifs-utils: Adapt lpt subsystem in libubifs
Adapt lpt subsystem(lpt.c,lprops.c,lpt_commit.c) in libubifs, compared
with linux kernel implementations:
1. Remove debug related functions(eg. dbg_chk_lpt_sz, dbg_chk_pnode),
some of debug functions are not needed by fsck, because fsck will
check fs in another way.
2. Remove some functions(eg. ubifs_create_dflt_lpt) which won't be used
in fsck/mkfs.
3. Adapt do_calc_lpt_geom() to mkfs/fsck situations.
4. Adapt calc_dflt_lpt_geom to support the mkfs tool, and export it.
5. Adapt ubifs_create_lpt() according to create_lpt(mkfs), make sure that
the height of created lpt is right.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:58 +0000 (16:36 +0800)]
ubifs-utils: Adapt io.c in libubifs
Adapt io.c in libubifs, compared with linux kernel implementations:
1. Modify io related functions(eg. ubifs_leb_read/ubifs_leb_write,
etc.), adapt them with userspace io functions lseek/read/write.
2. Remove some functions(eg. record_magic_error, ubifs_bg_wbufs_sync)
which won't be used in fsck/mkfs.
3. Replce ubifs_errc with ubifs_err, because there will be no SB_SILENT
options in mkfs/fsck.
4. Initiate wbuf->size as c->max_write_size.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:57 +0000 (16:36 +0800)]
ubifs-utils: Adapt super.c in libubifs
Adapt super.c in libubifs, compared with linux kernel implementations:
1. It contains all definitions in common/super.c(Message printing
functions are replaced with linux kernel styles).
2. Remove some functions(eg. ubifs_iget, ubifs_dirty_inode) which won't
be used in fsck/mkfs.
3. Remove unused variables initialization in some functions(eg.
init_constants_early, init_constants_sb).
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
Zhihao Cheng [Mon, 11 Nov 2024 08:36:56 +0000 (16:36 +0800)]
ubifs-utils: Adapt ubifs header file in libubifs
Adapt ubifs header file in libubifs, compared with linux kernel
implementations:
1. New header file contains all definitions in common/ubifs.h
2. Remove some structures(eg. ubifs_mount_opts, bu_info) and
functions(eg. ubifs_sync_wbufs_by_inode, ubifs_jnl_XXX) which
won't be used in fsck/mkfs.
3. Modify some authentication related functions, keep functions
that mkfs may need, other functions are defined as empty.
4. Move ubifs message printing functions(ubifs_err/warn/msg) which
are originally defined in misc.c into ubifs.h.
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by: David Oberhollenzer <david.oberhollenzer@sigma-star.at>
projects / mtd-utils.git / log