diff --git a/src/content/docs/blog/2020/08/16/stage2-complete/index.md b/src/content/docs/blog/2020/08/16/stage2-complete/index.md index 318d2fb5..74e33140 100644 --- a/src/content/docs/blog/2020/08/16/stage2-complete/index.md +++ b/src/content/docs/blog/2020/08/16/stage2-complete/index.md @@ -27,6 +27,8 @@ For cross-compilation, we'll be relying on `qemu-static` to complete `2.5` and ` This is a major milestone for the project, as it is an early indication that we're self hosting. + + ## Multiple Architecture Support At this point in time, we now have build support for two targets: `x86_64` and `ARMV8a`. Our intent is to support `haswell` and above, or `zen` and above, for the x86_64 target. diff --git a/src/content/docs/blog/2020/08/20/stage3-progress/index.md b/src/content/docs/blog/2020/08/20/stage3-progress/index.md index 67b8892e..d7ab3b52 100644 --- a/src/content/docs/blog/2020/08/20/stage3-progress/index.md +++ b/src/content/docs/blog/2020/08/20/stage3-progress/index.md @@ -26,6 +26,8 @@ We're going to start working on a project to flesh out the `musl` runtime with s These will be maintained over at our [GitHub](https://github.com/serpent-os/libc-support) + + ## libwildebeest As a project we strive for correctness in the most pragmatic way. Some software, such as systemd, is heavily reliant on GNU GCC/GLibc extensions. In some software there are feasible alternatives when using `musl`, however in a good number of cases, functionality required to build certain diff --git a/src/content/docs/blog/2020/08/28/looking-stage4-in-the-eye/index.md b/src/content/docs/blog/2020/08/28/looking-stage4-in-the-eye/index.md index d42a33b2..30d4598c 100644 --- a/src/content/docs/blog/2020/08/28/looking-stage4-in-the-eye/index.md +++ b/src/content/docs/blog/2020/08/28/looking-stage4-in-the-eye/index.md @@ -16,6 +16,8 @@ Well, we've made an awful lot of progress in these last few days. It wasn't that Our [libc-support](https://github.com/ikeycode/libc-support) project has been growing, thanks primarily to the contributions of [Jouni Roivas](https://github.com/jroivas). We now have initial working versions of `getent` and `getconf`. The `getconf` program is considered feature-complete for our current requirements, and the focus is now on cleaning up `getent` making it more modular and easy to maintain in the long run. + + ## libwildebeest We began work on [libwildebeest](https://github.com/ikeycode/libwildebeest) to quickly unlock building `systemd`. Remember, our ambition with this project is to provide a sane, centralised way of maintaining source compatibility with various projects that rely on features currently only available in the GNU toolchain + runtime. This is our alternative vision to patching every single package that fails to build against our LLVM+musl toolchain, ensuring our work scales. diff --git a/src/content/docs/blog/2020/09/07/stage3-complete/index.md b/src/content/docs/blog/2020/09/07/stage3-complete/index.md index 8c3329e1..57565ce3 100644 --- a/src/content/docs/blog/2020/09/07/stage3-complete/index.md +++ b/src/content/docs/blog/2020/09/07/stage3-complete/index.md @@ -22,6 +22,8 @@ Moss does not aim to be a next generation package management solution, it instea In order to complete `stage3`, we provided linker stubs in `libwildebeest` to ensure we can build. Obviously this introduces runtime errors in systemd and our `stage3` isn't bootable, just chrootable. This will be resolved when systemd is properly packaged by moss in `stage4`. + + ## Docker Image We now have a test container available on Docker Hub. You can install and run the simple bash environment by executing the following command with a Docker-enabled user: diff --git a/src/content/docs/blog/2020/09/10/defining-moss/index.md b/src/content/docs/blog/2020/09/10/defining-moss/index.md index d208649e..89f589b9 100644 --- a/src/content/docs/blog/2020/09/10/defining-moss/index.md +++ b/src/content/docs/blog/2020/09/10/defining-moss/index.md @@ -27,6 +27,8 @@ This is typically achieved using a so-called `A/B switch`, which is what we will Primarily we'll update a single `/usr` symlink to point to the current OS image, with `/` being a read-only faux rootfs, populated with established mountpoints and symlinks. Mutation will be possible only via `moss` transactions, or in world-writable locations (`/opt`, `/usr/local`, `/etc` ...) + + ## Deduplication The moss binary package format will be deduplicated in nature, containing hash-keyed blobs in an `zstd` compressed payload. Unique blobs will be stored in a global cache, and hard-linked into their final location (i.e. `/moss/root/$number/usr/...`) to deduplicate the installed system too. This allows multiple system snapshots with minimal overhead, and the ability to perform an offline rollback. diff --git a/src/content/docs/blog/2020/09/20/moss-format-defined/index.md b/src/content/docs/blog/2020/09/20/moss-format-defined/index.md index f427732c..7420384f 100644 --- a/src/content/docs/blog/2020/09/20/moss-format-defined/index.md +++ b/src/content/docs/blog/2020/09/20/moss-format-defined/index.md @@ -50,6 +50,8 @@ The `Meta` payload consists of a number of records, each with strongly defined t ubyte[1] padding = 0; ``` + + ## TLDR that for me... Binary format that is self deduplicating at several layers, permitting fast transactional operations. diff --git a/src/content/docs/blog/2020/09/21/source-format-defined/index.md b/src/content/docs/blog/2020/09/21/source-format-defined/index.md index 85ec10a7..f257986a 100644 --- a/src/content/docs/blog/2020/09/21/source-format-defined/index.md +++ b/src/content/docs/blog/2020/09/21/source-format-defined/index.md @@ -39,6 +39,8 @@ We've opted to group "Package Definition" behaviour into core structs, which are rundeps: Different rundeps ``` + + ## Multiple architecture support In keeping with the grouping behaviour, we're baking multiple architecture configurations into the YML file. A common issue encountered with the older format was how to handle `emul32`: diff --git a/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.md b/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.md index fdaabaec..d4822a98 100644 --- a/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.md +++ b/src/content/docs/blog/2020/09/22/results-of-the-experiment/index.md @@ -19,6 +19,8 @@ It turns out that so far the experiment has been successful, and being forkable ![Toolchain bootstrap](./Featured.webp "Toolchain bootstrap") + + ## An open community Over the past 2 months we've transformed from set of ideas into a transparent, community-first organisation with clear leadership and open goals. I've stepped into the Benevolent Dictator For Life position, and Peter has taken on daily responsibilities for the project running. Aydemir is now our treasurer on Open Collective[^1], and many individuals contribute to our project. diff --git a/src/content/docs/blog/2020/10/11/build-system-functional/index.md b/src/content/docs/blog/2020/10/11/build-system-functional/index.md index 1f15454c..00f390ac 100644 --- a/src/content/docs/blog/2020/10/11/build-system-functional/index.md +++ b/src/content/docs/blog/2020/10/11/build-system-functional/index.md @@ -13,7 +13,6 @@ Wow, has it been a hectic few weeks, and it definitely shows: last time we blogg ![Look at all the buildiness](./Featured.webp "Look at all the buildiness") - ## It lives! We're now able to build **the majority** of a moss package. Notice I've made a distinction there. So, we're able to run the build instructions, and use all the metadata, configuration and macros available. The only thing we're not actually doing is dumping the built files into a binary package. @@ -33,6 +32,7 @@ We're able to do the build-system part _rather well_, now. Right now we support - Recursive macro support in build scripts, defined on per-profile level - Architecture specific profiles support in stone.yml, i.e. `profiles -> aarch64 -> setup:` to avoid if/else spaghetti. + ## Next on the agenda diff --git a/src/content/docs/blog/2021/02/08/cleanups-complete/index.md b/src/content/docs/blog/2021/02/08/cleanups-complete/index.md index 2e5c06aa..f94407eb 100644 --- a/src/content/docs/blog/2021/02/08/cleanups-complete/index.md +++ b/src/content/docs/blog/2021/02/08/cleanups-complete/index.md @@ -27,6 +27,8 @@ As it now stands, 1 lint issue (a non-issue, really) exists across all 4 project Granted, cleanups aren't all that sexy. Peter has been updating many parts of the bootstrap project, introducing systemd 247.3, LLVM 11.0.1, etc. We now have all 3 stages building correctly again for the x86_64 target. + + ## And then.... Yikes, tough audience! So we've formed a new working TODO which will make its way online as a public document at some point. The first stage, cleanups, is done. Next is to restore feature development, working specifically on the extraction and install routines. From there, much work and cadence will be unlocked, allowing us to work towards an MVP. diff --git a/src/content/docs/blog/2021/02/09/unlocking-moss/index.md b/src/content/docs/blog/2021/02/09/unlocking-moss/index.md index e321b81e..8c6a4266 100644 --- a/src/content/docs/blog/2021/02/09/unlocking-moss/index.md +++ b/src/content/docs/blog/2021/02/09/unlocking-moss/index.md @@ -21,6 +21,8 @@ Some patching has taken place to prevent using `ld.gold` and instead use `lld` t ![A wild LDC appears](./Featured.webp "A wild LDC appears") + + ## Reworking the Payload implementation Originally our prototype moss format only contained a `ContentPayload` for files, and a `MetaPayload` for metadata, package information, etc. As a result, we opted for simple structs, basic case handling and an iterable `Reader` implementation. diff --git a/src/content/docs/blog/2021/02/17/moss-format-read-write-support/index.md b/src/content/docs/blog/2021/02/17/moss-format-read-write-support/index.md index 55c72552..286e3d18 100644 --- a/src/content/docs/blog/2021/02/17/moss-format-read-write-support/index.md +++ b/src/content/docs/blog/2021/02/17/moss-format-read-write-support/index.md @@ -25,6 +25,8 @@ We've removed the old enumeration approach from the `Reader` class, instead requ Right now we can read and write our `MetaPayload` from and to the stream, allowing us to encode & decode the metadata associated with the package, with strong type information (i.e. `Uint64`, `String`, etc.) + + ## Next Steps We need to restore the `IndexPayload`, `LayoutPayload` and `ContentPayload` definitions. The first two are simply **data** payloads and will largely follow the design of the newly reimplemented `MetaPayload`. Then we restore `ContentPayload` support, and this will allow the next steps: unpack, install. diff --git a/src/content/docs/blog/2021/03/16/optimising-package-distribution/index.md b/src/content/docs/blog/2021/03/16/optimising-package-distribution/index.md index db4d8cd1..b6d26853 100644 --- a/src/content/docs/blog/2021/03/16/optimising-package-distribution/index.md +++ b/src/content/docs/blog/2021/03/16/optimising-package-distribution/index.md @@ -21,6 +21,8 @@ Minimising the size of updates is particularly valuable where files are download From the users point of view, minimising file size and upgrade time are important priorities, but for a developer, the speed at which packages are created and indexed is vital to progression. Deltas are different to packages in that they aren't required immediately, so there's minimal impact in taking longer to minimise their size. By getting deltas right, we can trade-off the size of packages to speed up development, while users will not be affected and fetch only size optimised deltas. + + ## Test Case - QtWebEngine QtWebEngine provides a reasonable test case where the package is a mix of binaries, resources and translations, but above average in size (157.3MB uncompressed). The first trade-off for speed over size has already been made by incorporating `zstd` in moss over `xz`, where even with max compression `zstd` is already 5.6% larger than using `xz`. This is of course due to the amazing decompression speeds where `zstd` is magnitudes faster. diff --git a/src/content/docs/blog/2021/03/20/moss-unlocked/index.md b/src/content/docs/blog/2021/03/20/moss-unlocked/index.md index bb62d7f3..ad4fe6d1 100644 --- a/src/content/docs/blog/2021/03/20/moss-unlocked/index.md +++ b/src/content/docs/blog/2021/03/20/moss-unlocked/index.md @@ -18,6 +18,8 @@ Oh yeah, that totally happened. So, we can now successfully build moss packages Admittedly the code is super rough within `moss` and somewhat proof of concept, however we're able to extract the contents of the moss archive and rebuild the layout on disk. + + ## What makes extraction difficult? Well, quirky new format for one. A moss archive currently consists of 4 "payloads", or major sections: diff --git a/src/content/docs/blog/2021/05/18/let-there-be-databases/index.md b/src/content/docs/blog/2021/05/18/let-there-be-databases/index.md index b553f544..cb5b9fc2 100644 --- a/src/content/docs/blog/2021/05/18/let-there-be-databases/index.md +++ b/src/content/docs/blog/2021/05/18/let-there-be-databases/index.md @@ -20,6 +20,8 @@ So far we've implemented a "StateMetaDB", which stores metadata on every recorde Next on the list is the LayoutsDB (file manifests) and the CacheDB, for recording refcounts of every cached file in the OS pool. + + ## Integration with Serpent ECS An interesting trial we're currently implementing is to hook the DB implementation up to our Entity Component system from the Serpent Engine, in order to provide fast, cache coherent, in memory storage for the DB. It's implemented using many nice DLang idioms, allowing the full use of `std.algorithm` APIs: diff --git a/src/content/docs/blog/2021/08/02/initial-performance-testing/index.md b/src/content/docs/blog/2021/08/02/initial-performance-testing/index.md index 62b8c224..c0457566 100644 --- a/src/content/docs/blog/2021/08/02/initial-performance-testing/index.md +++ b/src/content/docs/blog/2021/08/02/initial-performance-testing/index.md @@ -26,6 +26,8 @@ It is important to note that in this comparison the Host distro has pulled in so Based on the results during testing, the performance of `clang` in Serpent OS still has room to improve and was just a quick tuning pass. At stages where I would have expected to be ahead already, the compile performance was only equal (but `cmake` and `configure` were still well ahead). + + ## GCC Matters Too! While `clang` is the default compiler in Serpent OS, there may be instances where the performance is not quite where it could be. It is common to see software have more optimized code paths where they are not tested with `clang` upstream. As an example, here's a couple of patches in flac ([1](https://github.com/xiph/flac/commit/67ea8badadd3e63b8e8af5fe837d075104569330), [2](https://github.com/xiph/flac/commit/d4a1b345dd16591ff6f17c67ee519afebe2f9792)) that demonstrate this being improved. Using `benchmarking-tools`, it is easy to see where `gcc` and `clang` builds are running different functions via `perf` results. diff --git a/src/content/docs/blog/2021/08/03/moss-db-progress/index.md b/src/content/docs/blog/2021/08/03/moss-db-progress/index.md index 8b989142..224cbb75 100644 --- a/src/content/docs/blog/2021/08/03/moss-db-progress/index.md +++ b/src/content/docs/blog/2021/08/03/moss-db-progress/index.md @@ -54,6 +54,8 @@ And this is the new, shmexy way: It's quite easy to see the new API lends itself robustly to our needs, so that we may implement stateful, strongly typed databases for moss. + + ## Next Steps Even though some APIs in `moss-db` may still be lacking (remove, for example) we're happy that it can provide the foundation for our next steps. We now need to roll out the new `StateDB`, `MetaDB` and `LayoutDB`, to record system states, package metadata, and filesystem layout information, respectively. diff --git a/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.md b/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.md index 1183fc46..76d4f624 100644 --- a/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.md +++ b/src/content/docs/blog/2021/08/10/a-rolling-boulder-gathers-no-moss/index.md @@ -26,6 +26,8 @@ Well, persistence really. Thanks to `RocksDB` and our new `moss-db` project, we Additionally, we remember the layouts within the LayoutDB so that we can eventually recreate said layout on disk. + + ## Precaching Before we actually commit to an install, we try to precache all of the stone files in our pool. So we unpack the content payload ("megablob"), split it into various unique files in the pool ready for use. At this point we also record the Layouts, but do not "install" the package to a system root. diff --git a/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.md b/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.md index 2fe923eb..459e1170 100644 --- a/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.md +++ b/src/content/docs/blog/2021/08/25/unpacking-the-build-process-1/index.md @@ -26,6 +26,8 @@ Some key parts to setting up the build environment: While the focus of early optimization work has been on build time performance, there's more overhead to creating packages time than simply compiling code. Now the compiler is in a good place, we can explore the rest of the build process. + + ## Packaging is More than just Compile Time There's been plenty of progress in speeding up the creation of the build environment such as parallel downloads to reduce connection overhead and using `zstd` for the fast decompression of packages. But there's more that we can do to provide an optimal experience to our packagers. diff --git a/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.md b/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.md index 05a9effe..722286a9 100644 --- a/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.md +++ b/src/content/docs/blog/2021/09/20/unpacking-the-build-process-2/index.md @@ -22,6 +22,8 @@ The steps for compiling code are generally quite straight-forward: This will build software compiled against packages installed on your system. It's a bit more complicated when packaging as we first set up an environment to compile in (Part 1). But even then you have many choices to make and each can have an impact on how long it takes to compile the code. Do you build with Link Time Optimizations (LTO) or Profile Guided Optimizations (PGO), do you build the package for performance or for the smallest size? Then there's packages that benefit considerably from individual tuning flags (like `-fno-semantic-interposition` with `python`). With so many possibilities, `boulder` helps us utilize them through convenient configuration options. + + ## What Makes boulder so Special? As I do a lot of packaging and performance tuning, `boulder` is where I spend most of my time. Here are some key features that `boulder` brings to make my life easier. diff --git a/src/content/docs/blog/2021/10/04/optimal-file-locality/index.md b/src/content/docs/blog/2021/10/04/optimal-file-locality/index.md index df1bcbce..7c39e63c 100644 --- a/src/content/docs/blog/2021/10/04/optimal-file-locality/index.md +++ b/src/content/docs/blog/2021/10/04/optimal-file-locality/index.md @@ -18,6 +18,8 @@ To test the efficiency our content payload, the natural choice was to compare it Of course it does (otherwise it would be a pretty crappy blog post!). When extracting a `.stone` package it creates two directories, `mossExtract` where the sha256sum named files are stored and `mossInstall` where those files are hardlinked to their full path name. The first day I created the tarball from `mossInstall` and the second day I realised that creating the tarball from `mossExtract` would provide the closest match to the content payload since it was a direct comparison. When compressing the tarballs to match the `.stone` compression level, the tarball compressed from `mossInstall` was 10% smaller, despite the uncompressed tarball being slightly larger. + + ## Compression Wants to Separate Apples and Oranges In simplistic terms, the way compression works is comparing data that it's currently reading versus data that it's read earlier in the file. `zstd` has some great options like `--long` that increases the distance in which these matches can be made at the cost of increased memory use. To limit memory use while making compression and decompression fast, it takes shortcuts that reduce the compression ratio. For optimal compression, you want files that are most similar to each other to be as close as possible. You won't get as many matches from a text file to an ELF file as you would from a similar looking text file. diff --git a/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.md b/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.md index af6b04d5..c15eb2ae 100644 --- a/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.md +++ b/src/content/docs/blog/2021/11/05/performance-corner-faster-builds-smaller-packages/index.md @@ -27,6 +27,8 @@ Payload: Content [Records: 2550 Compression: Zstd, Savings: 81.46%, Size: 236.72 The build time is a little high, but a lot of that is due to a slow compiler on the host machine. But analysing and producing packages was also taking a lot longer than it needed to. + + ## The Death of moss-jobs in boulder In testing an equivalent build outside of `boulder`, the build stages were about 5% faster. Testing under `perf`, the jobs system was a bit excessive for the needs of `boulder`, polling for work when we already know the times when parallel jobs would be useful. Removing `moss-jobs` allowed for simpler codepaths using multiprocessing techniques from the core language. This work is integrated in `moss-deps` and the excess overhead of the build has now been eliminated. diff --git a/src/content/docs/blog/2021/11/23/it-all-depends/index.md b/src/content/docs/blog/2021/11/23/it-all-depends/index.md index d8adf30d..7f4cec88 100644 --- a/src/content/docs/blog/2021/11/23/it-all-depends/index.md +++ b/src/content/docs/blog/2021/11/23/it-all-depends/index.md @@ -20,6 +20,8 @@ As such we encode dependencies into our (endian-aware, binary) format which is t Lastly, we'll always ensure there is no possibility for "partial update" woes. With these considerations, we have no need to support `>=` style dependencies, and instead rely on strict design goals and maintainer responsibility. + + ## First and foremost! The rapid move we're enjoying from concept, to prototype, and soon to be fully fledged Linux distribution, is only possible with the amazing community support. The last few months have seen us pull off some amazing feats, and we're now executing on our first public milestones. With your help, more and more hours can be spent getting us ready for release, and would probably help to insulate my shed office! (Spoiler: its plastic and electric heaters are expensive 😀) diff --git a/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.md b/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.md index 54d16d72..c9ea00d5 100644 --- a/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.md +++ b/src/content/docs/blog/2021/12/14/performance-corner-small-changes-pack-a-punch/index.md @@ -10,7 +10,6 @@ copyright: "Copyright © 2021 aerynOS Developers" Here we have another round of changes to make packages smaller and show just how much we care about performance and efficiency! Today we are focusing mainly on `moss-format` changes to reduce the size of its payloads. The purpose of these changes is to reduce the size of packages, DB storage for transactions and memory usage for `moss`. These changes were made just before we moved out of bootstrap, so we wouldn't have to rebuild the world after changing the format. Lets get started! - ## Squeezing Out the Last Blit of Performance `Blitting` in Serpent OS is the process of setting up a root, using hardlinks of files installed in the `moss` store to construct the system `/usr` directory. Some initial testing showed `buildPath` using about 3% of the total events when installing a package with many files. As a system is made up of over 100,000 files, that's a lot of paths that need to be calculated! Performance is therefore important to minimize time taken and power consumed. @@ -29,6 +28,8 @@ After: 12,814,283 ( 0.40%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath ``` + + ## Only Record What's Needed! Our `Index Payload` is used for extracting the `Content Payload` into its hashed file names. As it has one job (and then discarded), we could hone in on including only the information we needed. Previously an entry was a 32 byte key and storing a 33 byte hash. We have now integrated the hash as a `ubyte[16]` field, cut other unneeded fields so that we can fit the whole entry into 32 bytes. That's a 51% reduction in the size of the `Index Payload` and about 25% smaller when compressed. diff --git a/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.md b/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.md index 66679e97..4fa4562e 100644 --- a/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.md +++ b/src/content/docs/blog/2022/01/20/can-hardly-contain-myself-plus-a-bonus/index.md @@ -20,6 +20,8 @@ Containers have a multitude of uses for a Linux distro, but our immediate use ca The next step is integrating `moss-container` into `boulder`, so that builds become reproducible across machines, and makes it much easier for users to run builds on their host machines. + + ## moss Now Understands Repositories Previously (but not covered in the blogs) work was also done on `moss` so that it can understand and fetch `stone` packages from an online repo. This ties in nicely with the `moss-container` work and is a requirement for finishing up a proper build process for Serpent OS. We are now one step closer to having a full distribution cycle from building packages and pushing those packages as system updates! diff --git a/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.md b/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.md index d32ce5c2..ebcce10b 100644 --- a/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.md +++ b/src/content/docs/blog/2022/02/11/making-deltas-great-again-1/index.md @@ -16,6 +16,8 @@ I remember back in the early 2000s on Gentoo when someone set up a server to pro Today things are quite different. The case for deltas has reduced for users as internet speeds have increased. Shrinking 20MB off a package size may be a second reduction for some, but 10 seconds for others. The largest issue is that deltas have typically pushed extra computational effort onto the users computer in compensation for the smaller size. With a fast internet connection that cost is a real burden where deltas take longer to install than simply downloading the full package. + + ## What's so Bad About the Old Method? The previous idea of using the full payload for deltas was very efficient in terms of distribution, but required changes in how `moss` handles packages to implement it. Having the previous payload available (and being fast) means storing the old payloads on disk. This increases the storage requirements for the base system, although that can be reduced by compressing the payloads to reduce disk usage (but increasing CPU usage at install time). diff --git a/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.md b/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.md index 5e1a948b..daba90cc 100644 --- a/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.md +++ b/src/content/docs/blog/2022/04/05/relr-brings-smaller-files-more-performance/index.md @@ -27,6 +27,8 @@ Here's an example of the sections of `libLLVM-13.so` with and without `RELR`. He | | | | | Total | 44,853,987 | 42,966,764 | + + ## Smaller, But at What Cost? While most of the discussion about `RELR` is around the size savings, there's been very little in terms of the performance numbers of enabling `RELR`. For most things, it's not going to make a noticeable difference, as it should only really impact loading of binaries. There's one rule we have and that's to measure everything! We care about every little detail where many 1-2% improvements can add up to significant gains. diff --git a/src/content/docs/blog/2022/06/22/packaging-automation-next-steps/index.md b/src/content/docs/blog/2022/06/22/packaging-automation-next-steps/index.md index 70e446d1..1a2b7678 100644 --- a/src/content/docs/blog/2022/06/22/packaging-automation-next-steps/index.md +++ b/src/content/docs/blog/2022/06/22/packaging-automation-next-steps/index.md @@ -24,6 +24,8 @@ We prefer a self documenting approach that can be machine validated rather than ..Doesn't mean we can't make packaging **even easier** though. + + ## Build discovery Pointing boulder at an upstream source will perform a deep analysis of the sources to determine the build system type, build dependencies, metadata, licensing etc. Right now it's just getting ready to leave POC stage so it has a few warts, however it does have support for generating package skeletons for the following build systems: diff --git a/src/content/docs/blog/2022/07/10/infrastructure-update/index.md b/src/content/docs/blog/2022/07/10/infrastructure-update/index.md index 51643207..17220774 100644 --- a/src/content/docs/blog/2022/07/10/infrastructure-update/index.md +++ b/src/content/docs/blog/2022/07/10/infrastructure-update/index.md @@ -18,6 +18,8 @@ During the development of the Serpent OS tooling we've been exploring the possib In practice that required an encapsulation API written in D around the C API, and our own wrapping on top of that. Naturally, it resulted in a very allocation-heavy implementation that just didn't sit right with us, and due to the absurd complexity of RocksDB was still missing quite a few features. + + ## Enter LMDB We're now using the [Lightning Memory-Mapped Database](https://www.symas.com/lmdb) as the driver implementation for moss-db. In short, we get rapid reads, ACID transactions, bulk inserts, you name it. Our implementation takes advantage of multiple database indexes (`MDB_dbi`) in LMDB to partition the database into internal components, so that we can provide "buckets", or collections. These internal DBs are used for bucket mapping to permit a key-compaction strategy - iteration of top level buckets and key-value pairs within a bucket. diff --git a/src/content/docs/blog/2022/12/24/lift-off/index.md b/src/content/docs/blog/2022/12/24/lift-off/index.md index 249a8137..0cb11579 100644 --- a/src/content/docs/blog/2022/12/24/lift-off/index.md +++ b/src/content/docs/blog/2022/12/24/lift-off/index.md @@ -20,6 +20,8 @@ As mentioned, we spent 2 years working on tooling and process. That's .. well. K I deferred final enabling of the infrastructure until January to rectify the chicken/egg scenario whilst allowing us to grow a base of contributors and an actual distro to work with. We're in a good position with minimal blockers so no concern there. + + ### A real software collection This is our term for the classical "package repository". We're using a temporary collection right now to store all of the builds we produce. In keeping with the `Avalanche` requirements, this is the **volatile** software collection. Changes a lot, hasn't got a release policy. diff --git a/src/content/docs/blog/2023/04/21/snakes-on-a-boat/index.md b/src/content/docs/blog/2023/04/21/snakes-on-a-boat/index.md index e5cb8ab0..bae322dd 100644 --- a/src/content/docs/blog/2023/04/21/snakes-on-a-boat/index.md +++ b/src/content/docs/blog/2023/04/21/snakes-on-a-boat/index.md @@ -26,6 +26,8 @@ Oh no, no. The most practical angle to view this from is that Serpent OS is free It is therefore critical that we continue development, and find the best approach for both projects, to make the transition to Solus 5 as seamless as possible. To that end we will still need to produce ISOs and have an active community of users and testers. During this transition period, despite being two separate projects, we're both heading to a common goal and interest. + + ### Finances We have an exciting journey ahead for all of us, and there are many moving parts involved. Until we're at the point of mutual merge, we will keep the entities and billing separate. Thus, funds to the Solus [OpenCollective](https://opencollective.com/getsolus) are intended for use within Solus, whereas we currently use GitHub Sponsors[^1] for our own project needs. diff --git a/src/content/docs/blog/2023/09/06/oxidised-moss/index.md b/src/content/docs/blog/2023/09/06/oxidised-moss/index.md index 44afc2f3..6fe511c0 100644 --- a/src/content/docs/blog/2023/09/06/oxidised-moss/index.md +++ b/src/content/docs/blog/2023/09/06/oxidised-moss/index.md @@ -22,6 +22,8 @@ While we're quite happy to be an upstream for Linux distributions by way of rele With that said, I will still continue to use D in my own *personal* projects, and firmly believe that one day D will realise its true potential. + + ## Rewarding contributors Our priorities have shifted somewhat since the announcement of our shared venture with Solus, and we must make architectural decisions based on the needs of all stakeholders involved, including the existing contributor pool. Additionally, care should be taken to be somewhat populist in our choice of stacks in order to give contributors industry-relevant experience to add to their résumé (CV). diff --git a/src/content/docs/blog/2023/12/19/end-of-year-summary/index.md b/src/content/docs/blog/2023/12/19/end-of-year-summary/index.md index 476b74f7..21a35b01 100644 --- a/src/content/docs/blog/2023/12/19/end-of-year-summary/index.md +++ b/src/content/docs/blog/2023/12/19/end-of-year-summary/index.md @@ -24,6 +24,8 @@ Firstly, a quote from Cory: Chances are you know tarkah for his contributions to [iced-rs](https://github.com/iced-rs/iced), and to [Solus](https://getsol.us). His contributions to the [moss-rs](https://github.com/serpent-os/moss-rs) project have been absolutely critical, and his patience essential as we got ourselves quickly up to speed with Rust. It is entirely fair to say that our Rust efforts would not have been possible without him! + + ## Rust Progress I think it is fair to say that people collectively facepalmed when we announced our plans to [adopt Rust](/blog/2023/09/06/oxidised-moss/) - assuming this would be a huge set back. We're really happy to report this is **not** the case, and we've made tremendous progress in this area. diff --git a/src/content/docs/blog/2024/01/19/january-updates/index.md b/src/content/docs/blog/2024/01/19/january-updates/index.md index dfe53ccf..0442be75 100644 --- a/src/content/docs/blog/2024/01/19/january-updates/index.md +++ b/src/content/docs/blog/2024/01/19/january-updates/index.md @@ -26,6 +26,8 @@ To reiterate a critical point, we require that our contributions are attributed Additionally we picked MPL-2.0 over Zlib due to some distinct advantages in relation to patent trolls. In combination we believe both the project and our users will have the greatest protection from patent trolls and fiscally-motivated relicensing. + + ## As I Raise My Trigger Finger As you may recall from the last post, we announced our intention to add system triggers to moss. We also stated that our triggers must run in **isolation** thanks to the unique architecture of our package management. After much consideration, we've landed on a strategy that will work for us. diff --git a/src/content/docs/blog/2024/02/29/end-of-feb-update/index.md b/src/content/docs/blog/2024/02/29/end-of-feb-update/index.md index 6e92560d..dfbc5ad8 100644 --- a/src/content/docs/blog/2024/02/29/end-of-feb-update/index.md +++ b/src/content/docs/blog/2024/02/29/end-of-feb-update/index.md @@ -22,6 +22,8 @@ Just like `moss` it makes use of `tokio` for async right where we need it, and i ![Breakpoints, in recipes](BoulderBreakpoints.webp "Breakpoints in recipes") + + ## Triggers, integrated! After much discussion we finally integrated support for *triggers* in `moss`. These are actions that are executed at different stages of package management operations to finalize or "bake" some state based on the unified components of the installation. diff --git a/src/content/docs/blog/2024/03/31/the-march-of-progress/index.md b/src/content/docs/blog/2024/03/31/the-march-of-progress/index.md index 0b3dfba2..67376aae 100644 --- a/src/content/docs/blog/2024/03/31/the-march-of-progress/index.md +++ b/src/content/docs/blog/2024/03/31/the-march-of-progress/index.md @@ -16,6 +16,8 @@ Despite a brief excursion out of the country for a first-in-a-lifetime holiday, OK, we all know you want to know about the baremetal stuff, but c'mon, check out this changeset! + + ### Boulder We're officially at a point where the Rust boulder implementation has become our "blessed" build tool. This is actually a crate within the `moss` repository, allowing the two tools to share huge chunks of code. Our focus over the past few weeks has been to ensure we can drop the tool into our existing build network and "just work", while making it significantly faster. diff --git a/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.md b/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.md index 977175a1..d7b72691 100644 --- a/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.md +++ b/src/content/docs/blog/2024/04/30/calm-before-the-storm/index.md @@ -22,6 +22,8 @@ The boot management code is currently landing over the coming days, and will be We acknowledge this is an exchange of the pursuit of perfection for a pragmatic rapid iteration cycle and believe the best way forward is to open the doors wide, fixing issues on the job. The initial system will likely be rough and buggy, but it is an official starting point from which we can all build Serpent OS together. + + ## Moss + Boulder Our Rust tooling has now been in "production" use in our development environment and build servers for the last month without any major hiccup. We have recently discovered some minor issues in our `emul32` (32-bit) package generation which are being resolved, chiefly our move from the improperly named ISA `x86`, now updated to `386` affecting 32-bit dependency chains. diff --git a/src/content/docs/blog/2024/08/01/prealpha0-released/index.md b/src/content/docs/blog/2024/08/01/prealpha0-released/index.md index 304d1a25..36d87a4e 100644 --- a/src/content/docs/blog/2024/08/01/prealpha0-released/index.md +++ b/src/content/docs/blog/2024/08/01/prealpha0-released/index.md @@ -26,6 +26,8 @@ To detract from casual use it is necessary to manually partition the disk before You will need a working network connection, so make sure you're connected before starting the installer! + + ## Technologies involved This will of course appear to be a very rough (crap) prealpha ISO. Underneath the surface it is using the [moss](https://github.com/serpent-os/moss) package manager, our very own package management solution written in Rust. Quite simply, every single transaction in moss generates a new filesystem tree (`/usr`) in a staging area as a full, stateless, OS transaction. diff --git a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.md b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.md index 883e10ce..5fa9442c 100644 --- a/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.md +++ b/src/content/docs/blog/2024/12/23/serpent-os-enters-alpha/index.md @@ -23,6 +23,8 @@ Lastly, you can support the project by becoming a sponsor. Your support helps co + + ## What is Serpent OS? Serpent OS is a heavily engineering-led Linux distribution seeking to redefine *how* we distribute Linux. It is a stateless OS that leverages atomic updates, cutting-edge tooling, and rock-solid reliability to deliver a safe and efficient system. Built by industry veterans with decades of experience, Serpent OS represents the next evolution in Linux distributions. diff --git a/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.md b/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.md index 183d1fac..a78fd8ec 100644 --- a/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.md +++ b/src/content/docs/blog/2024/12/29/alpha-refresh-available/index.md @@ -17,6 +17,8 @@ In response to feedback from the community, we're issuing a refresh of the Alpha ![Fractional scaling, much loved](./featured-background.webp "Fractional scalling now enabled by default") + + ## Clarification on moss boot management Frequently in Team Serpent we joke that `moss` is the "systemd of package managers". It actually does a whole lot more than is easily visible.. for example: diff --git a/src/content/docs/blog/2024/12/31/investing-in-the-future/index.md b/src/content/docs/blog/2024/12/31/investing-in-the-future/index.md index a51ef735..a2ec41ec 100644 --- a/src/content/docs/blog/2024/12/31/investing-in-the-future/index.md +++ b/src/content/docs/blog/2024/12/31/investing-in-the-future/index.md @@ -19,6 +19,8 @@ Since the Alpha 1 ISO (`0.24.5`), we've fixed a number of issues in relation to We've been working hard to keep the cadence up, and we're now at a point where we can deliver new ISOs on a regular basis. Our main focus now is the tooling, in enabling a far more automatic delivery pipeline. The vision is having automated pull requests generated for package updates, automatically handling changes in dependencies and ABI, and ordered by build tier. This will significantly reduce the maintainer burden, in conjunction with planned security monitoring. + + ## Short term deliverables In the short term we're going to land the following features: diff --git a/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.md b/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.md index fd6538e7..046e28a1 100644 --- a/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.md +++ b/src/content/docs/blog/2025/01/04/offline-rollbacks-enabled/index.md @@ -14,8 +14,6 @@ If a picture tells a thousand words... well then this video .. Yknow what, sod i This has already landed in our volatile repository, so simply `sudo moss sync -u` and grab the latest updates. Any subsequent transactions will automatically generate the boot entries for you, no intervention required. Or mounting. Or anything. Just update and go. Srsly. - - :::note[Missing YouTube video] There used to be a YouTube video at this part of the blog, however our YouTube channel was unfortunately deleted outside of our control and we no longer have access to the video. ::: @@ -24,6 +22,8 @@ We actually saw this as a golden standard for the feature, as it's a real-world The intent is to ensure a user can quickly revert to the last transaction that worked in the instance an update goes awry, without needing to boot into a rescue environment or live CD. + + ### Like what we're doing? Developing a project with the scope of Serpent OS is no small feat. In order to maintain our current cadence, and to land huge features like the offline rollback, or impending system model work, we need your support. I'm as happy as the next person to return to the job market once my medical conditions ease up, but the simple matter is this will greatly slow down our recent progress. diff --git a/src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.md b/src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.md index 7caf5b30..014b7c5b 100644 --- a/src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.md +++ b/src/content/docs/blog/2025/11/04/Musings on Inode Watchers and Atomic Live Upgrades/index.md @@ -23,6 +23,8 @@ This explanation leaves out a couple of details such post-install triggers and b Compared to other atomic distributions on the market, the ability to live-update without needing to reboot is an important usability requirement for us, such that the user experience remains friendly to downstream users. + + ### Enter Inode Watchers Although quite a novel approach, allowing atomic updates of a live system leaves us with an interesting problem: After any moss transaction activating a new `/usr` tree, any running applications holding an underlying inode to a filesystem path will continue to watch the file in the now archived state.