Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

500 cleaned update #129

Closed
wants to merge 23 commits into from
Closed

Conversation

palainp
Copy link
Member

@palainp palainp commented Dec 26, 2023

This PR is a rebase against current main + updates some constants to be the same as in linux.

The bad news is that I'm currently in a dead end with mmap/munmap:
I first wanted to use malloc/posix_mem_align as it's available with dlmalloc as a mmap replacement, but going on that path led me to search how Ocaml uses mmap/munmap. If I understand correctly, Ocaml requests memory with mmap (more than necessary, maybe to ensure some alignments constraints?) and then uses munmap to "free" pages it won't use:

  • So we can't use free here as some pages haven't been allocated with malloc
  • If I noop munmap it works, but it leads to more memory usage/memory wasted than necessary (which isn't great as Ocaml forces an additional 2MB stack for each domains)

@palainp
Copy link
Member Author

palainp commented Dec 28, 2023

The last commit re-uses the bmap allocator from mirage-xen (https://github.com/mirage/mirage-xen/blob/main/lib/bindings/bmap.c) and permits to both mmap & munmap. So far it works with qubes mirage firewall (transmit traffic) and I have the same bandwith on an ADSL link (which isn't an indicator per se as the bandwidth is limited).

I haven't done any other performance tests yet (the bmap allocation is O(n) with n=2048 pages, and munmap is O(1)) because I'm more worried about having to set a magic constant ("mmap size", here 8MB) in the libc startup function.

@palainp palainp marked this pull request as draft December 28, 2023 21:14
@palainp
Copy link
Member Author

palainp commented Dec 31, 2023

So having an allocation for mmap leads to bad things, the Ocaml runtime runs out of memory at some point. So I'm switching to another memory allocation system:

  • the bitmap system for managing free/used pages
  • a binary buddy system for memory request smaller than a page

This currently works with tutorial/hello unikernel, but segfault with application/https and leave the CPU spinning with qubes-mirage-firewall. I've left debug trace to continue some work on that early 2024 :)

@palainp
Copy link
Member Author

palainp commented Jan 7, 2024

It seems to work now, qubes-mirage-firewall works and routes traffic :)

So far:

  • I have a very similar bandwidth with FTTH (I expected the simplest allocation algorithm to be slowest and maybe be a bottleneck, maybe with more heap memory (currently using 32MB) the O(n) complexity will shows some issues),
  • the Ocaml runtime uses a few extra MB at startup (13MB 5.1 vs 8MB with 4.14.1, 2MB is due to domain stack allocation, I haven't checked for the other 3) but memory usage is similar after a while (due to uncollected Cstruct bigarrays?),
  • I still need the manual GC calls with qubes-mirage-firewall

@palainp
Copy link
Member Author

palainp commented Mar 8, 2024

Closing this as @shym has been working on 5.2 integration and the allocator isn't well adapted.

@palainp palainp closed this Mar 8, 2024
@shym shym mentioned this pull request Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants