Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Malformed FAT entries mismatch #41

Closed
tgross35 opened this issue Jul 26, 2023 · 7 comments · Fixed by #58 · May be fixed by #44
Closed

Malformed FAT entries mismatch #41

tgross35 opened this issue Jul 26, 2023 · 7 comments · Fixed by #58 · May be fixed by #44

Comments

@tgross35
Copy link
Contributor

Is it possible that #1 regressed at some point? I am getting:

Malformed FAT (FAT has 30208 entries, but file has only 18279 sectors)

I intend to try this with a few different versions but have not yet had the opportunity
@ikrivosheev
Copy link
Contributor

@tgross35 hello! Could you send a file?

@tgross35
Copy link
Contributor Author

Er... Unfortunately not this specific one since it's a work thing, but I'm trying to get a minimal reproduction that I will send. The file is pretty large (>10MB) and is produced by Altium (not that that helps on its own).

@tgross35
Copy link
Contributor Author

Hey @ikrivosheev I never got a minified file but here are two header & footer dumps:

File 1, 9539584 bytes (9.1M), produces `io error: Malformed FAT (FAT has 30208 entries, but file has only 18631 sectors).` : 
00000000: d0cf 11e0 a1b1 1ae1 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 3e00 0300 feff 0900  ........>.......
00000020: 0600 0000 0000 0000 0000 0000 9200 0000  ................
00000030: ec46 0000 0000 0000 0010 0000 5a00 0000  .F..........Z...
00000040: b700 0000 8842 0000 0100 0000 5400 0000  .....B......T...
00000050: 8300 0000 5c01 0000 8301 0000 2802 0000  ....\.......(...
00000060: 8802 0000 0603 0000 8103 0000 0104 0000  ................
00000070: 8304 0000 0305 0000 9005 0000 0406 0000  ................
00000080: b706 0000 0107 0000 9e07 0000 0508 0000  ................
00000090: 8308 0000 0609 0000 8a09 0000 020a 0000  ................
000000a0: 820a 0000 030b 0000 810b 0000 0d0c 0000  ................
000000b0: 810c 0000 170d 0000 810d 0000 1a0e 0000  ................
000000c0: 840e 0000 020f 0000 810f 0000 0310 0000  ................
000000d0: 8210 0000 0711 0000 d411 0000 0212 0000  ................
000000e0: 8612 0000 0213 0000 2942 0000 2a42 0000  ........)B..*B..
000000f0: 2b42 0000 2c42 0000 2d42 0000 2e42 0000  +B..,B..-B...B..
00000100: 2f42 0000 3042 0000 3142 0000 3242 0000  /B..0B..1B..2B..
00000110: 3342 0000 3442 0000 3542 0000 3642 0000  3B..4B..5B..6B..
00000120: 3742 0000 3842 0000 3942 0000 3a42 0000  7B..8B..9B..:B..
00000130: 3b42 0000 3c42 0000 3d42 0000 3e42 0000  ;B..<B..=B..>B..
00000140: 3f42 0000 4042 0000 4142 0000 4242 0000  ?B..@B..AB..BB..
00000150: 4342 0000 4442 0000 4542 0000 4642 0000  CB..DB..EB..FB..
00000160: 4742 0000 4842 0000 4942 0000 4a42 0000  GB..HB..IB..JB..
00000170: 4b42 0000 4c42 0000 4d42 0000 4e42 0000  KB..LB..MB..NB..
00000180: 4f42 0000 5042 0000 5142 0000 5242 0000  OB..PB..QB..RB..
00000190: 5342 0000 5442 0000 5542 0000 5642 0000  SB..TB..UB..VB..
000001a0: 5742 0000 5842 0000 5942 0000 5a42 0000  WB..XB..YB..ZB..
000001b0: 5b42 0000 5c42 0000 5d42 0000 5e42 0000  [B..\B..]B..^B..
000001c0: 5f42 0000 6042 0000 6142 0000 6242 0000  _B..`B..aB..bB..
000001d0: 6342 0000 6442 0000 6542 0000 6642 0000  cB..dB..eB..fB..
000001e0: 6742 0000 6842 0000 6942 0000 6a42 0000  gB..hB..iB..jB..
000001f0: 6b42 0000 6c42 0000 6d42 0000 6e42 0000  kB..lB..mB..nB..
00000200: f8a6 0000 7c48 4541 4445 523d 5072 6f74  ....|HEADER=Prot

[... main file contents ...]

000769a0: 0000 0102 0000 0000 0100 0000 0000 0000  ................
000769b0: 0104 380a 00a0 004c ff00 0000 0005 4c53  ..8....L......LS
000769c0: 4e53 3002 3236 0003 7c26 7c00 2900 0001  NS0.26..|&|.)...
000769d0: 0200 0000 0001 0000 0000 0000 0001 0438  ...............8
000769e0: 0a00 a000 1aff 0000 0000 054f 5554 4d31  ...........OUTM1
File 2, 1387520 bytes (1.4M), produces `io error: Malformed FAT (FAT has 30208 entries, but file has only 14524 sectors).` : 
00000000: d0cf 11e0 a1b1 1ae1 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 3e00 0300 feff 0900  ........>.......
00000020: 0600 0000 0000 0000 0000 0000 1600 0000  ................
00000030: ea09 0000 0000 0000 0010 0000 2000 0000  ............ ...
00000040: 3a00 0000 feff ffff 0000 0000 1b00 0000  :...............
00000050: 8100 0000 1d01 0000 b201 0000 1002 0000  ................
00000060: 9002 0000 2003 0000 8203 0000 0204 0000  .... ...........
00000070: 8204 0000 0b05 0000 8105 0000 0406 0000  ................
00000080: 9606 0000 0907 0000 8207 0000 0308 0000  ................
00000090: 0709 0000 0809 0000 8109 0000 930a 0000  ................
000000a0: 940a 0000 ffff ffff ffff ffff ffff ffff  ................
000000b0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000c0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000d0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000e0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000f0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000100: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000110: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000120: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000130: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000140: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000150: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000160: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000170: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000180: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000190: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000001a0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000001b0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000001c0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000001d0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000001e0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000001f0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000200: 4534 0000 7c48 4541 4445 523d 5072 6f74  E4..|HEADER=Prot
00000210: 656c 2066 6f72 2057 696e 646f 7773 202d  el for Windows -

[... main file contents ...]

001529b0: 6d0a 0000 6e0a 0000 6f0a 0000 700a 0000  m...n...o...p...
001529c0: 710a 0000 720a 0000 730a 0000 740a 0000  q...r...s...t...
001529d0: 750a 0000 760a 0000 770a 0000 780a 0000  u...v...w...x...
001529e0: 790a 0000 7a0a 0000 7b0a 0000 7c0a 0000  y...z...{...|...
001529f0: 7d0a 0000 7e0a 0000 7f0a 0000 800a 0000  }...~...........
00152a00: 810a 0000 820a 0000 830a 0000 840a 0000  ................
00152a10: 850a 0000 860a 0000 870a 0000 880a 0000  ................
00152a20: 890a 0000 8a0a 0000 8b0a 0000 8c0a 0000  ................
00152a30: 8d0a 0000 8e0a 0000 8f0a 0000 900a 0000  ................
00152a40: 910a 0000 920a 0000 feff ffff fdff ffff  ................
00152a50: fdff ffff ffff ffff ffff ffff ffff ffff  ................
00152a60: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152a70: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152a80: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152a90: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152aa0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152ab0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152ac0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152ad0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152ae0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152af0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b00: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b10: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b20: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b30: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b40: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b50: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b60: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b70: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b80: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152b90: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152ba0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152bb0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152bc0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152bd0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152be0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00152bf0: ffff ffff ffff ffff ffff ffff ffff ffff  ................

Is this enough to go off of? I need to try to get a minified file that I can share still, but it only happens with fairly large files so it's not easy to make a sample.

@ikrivosheev
Copy link
Contributor

@tgross35 I reproduced your problem and found a sample. But it's malware file... Well, I will try to fix it.

@ikrivosheev
Copy link
Contributor

ikrivosheev commented Sep 11, 2023 •
edited
Loading

Well, now I understand what happens with python implementation. It skips continue parsing: https://github.com/decalage2/olefile/blob/master/olefile/olefile.py#L903C1-L903C92

@mdsteele what can we do with this problem? Maybe on Permissive validation mode we can truncate fat?

@ikrivosheev ikrivosheev mentioned this issue Sep 11, 2023
Open
@agashlin agashlin mentioned this issue Sep 27, 2024
Merged
@agashlin
Copy link
Contributor

I think I'm encountering the same issue with Altium as @tgross35, PR #58 should fix that. For an example file in the wild see the 19.7 MB MB-Q-RK3588-V10_Library.SchLib. Other than the zero padding in the DIFAT sector, the only strict violation in these files is a non-zero storage start sector, so I don't think fixing this specific issue will require allowing file truncation as in #44.

As I noted in the PR, a simpler fix would be to just truncate the DIFAT to header.num_fat_sectors, but given the number of other implementations this project needs to interact with I imagined it might be unreliable. So I made a more circumscribed change; Altium doesn't seem to use zero padding unless there is a DIFAT sector. I'll happily make the simpler truncation change if that would be ok with @mdsteele.

@agashlin
Copy link
Contributor

agashlin commented Sep 28, 2024 •
edited
Loading

Thanks very much for the quick turnaround! @tgross35, I wanted to check with you: You mentioned there was a 1.4M "File 2" with the issue, I wouldn't expect it then unless the FAT was excessively padded (as I did for the unit test). The error gives a sector count that indicates a file around 7M, and the header dump describes 0 DIFAT sectors so I wouldn't expect it to have this issue. My guess is that the dump, size, and error were from different files. But I could be mistaken; have you ever seen this error on a < 7M file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@ikrivosheev @tgross35 @agashlin