Public commit record
28/100 · Opaque
What the developer wrote
Authored by Niklas Dusenlund
Add bitbox-debug
✓ Subject identifies a change! No meaningful explanatory body
What changed, and why it matters
This commit adds a new internal debugging library called bitbox-debug and vendors several Rust dependencies (once_cell, portable-atomic, rtt-target). There is no indication in the commit message or diff that this fixes a security vulnerability. It appears to be a feature/infrastructure addition for firmware debugging.
Recommended action
No security action required. Treat as normal feature commit. If debug interfaces are exposed in production firmware, review the bitbox-debug crate separately for information disclosure risks, but that is outside the scope of this commit diff.
Security signals we found
No strong security signals were identified.
Evidence from the diff
The commit introduces src/rust/bitbox-debug, a new Rust crate, and vendors once_cell 1.21.4, portable-atomic 1.9.0, and rtt-target as dependencies. It updates Cargo.toml/Cargo.lock and adds a logging utility. The changes are additive and relate to debug tooling. No security-relevant bug fixes, privilege changes, or attack surface expansion beyond adding a debug crate are visible in the supplied diff.
Changed components
src/rust/bitbox-debugsrc/rust/util/src/log.rsexternal/vendor/once_cellexternal/vendor/portable-atomicexternal/vendor/rtt-targetInspect captured patch +13878 / −5749
### external/vendor/once_cell/.cargo-checksum.json
@@ -0,0 +1 @@
+{"files":{".cargo_vcs_info.json":"abe28464063d16bc9b0ebaad2b1682fc1a8052b0968bb4277f6b265844822c67",".github/workflows/ci.yaml":"b1c9b12babf51fcaa7de2adc737c91e882da240cf847becf380141cec320cf2d","CHANGELOG.md":"c104a8ec44b9087539185006e8187a8412883b902318e0a0a807a23ac756a070","Cargo.lock":"25438daaf9e812ea9c506823a9030c708b38e0a664039c91f4d16b05d0444dd4","Cargo.toml":"87e5b7ee438abdef12e46386e86370f54ad70cabdcf949556b9ef012a9da9587","Cargo.toml.orig":"cce5bf836e33ad0e9adfb6e900d432371af2ca3861da45b719a0fce716ea810f","LICENSE-APACHE":"a60eea817514531668d7e00765731449fe14d059d3249e0bc93b36de45f759f2","LICENSE-MIT":"23f18e03dc49df91622fe2a76176497404e46ced8a715d9d2b67a7446571cca3","README.md":"2331182c8b5a6971fd0d04a0ca711d5839e93b3de6b2003108940a8c93850aaf","bors.toml":"ebd69f714a49dceb8fd10ebadfea6e2767be4732fdef49eddf6239151b4bc78c","examples/bench.rs":"1597a52529f75d6c5ad0b86759a775b1d723dfa810e2016317283b13594219da","examples/bench_acquire.rs":"9f4912ca262194cb55e893c33739c85c2f4868d07905b9dd3238552b6ce8a6e4","examples/lazy_static.rs":"8bca1b264da21eceb1ccaf30477fc941bc71bedd030f1c6982ed3a7804abfb4f","examples/reentrant_init_deadlocks.rs":"ff84929de27a848e5b155549caa96db5db5f030afca975f8ba3f3da640083001","examples/regex.rs":"4a2e0fb093c7f5bbe0fff8689fc0c670c5334344a1bfda376f5faa98a05d459f","examples/test_synchronization.rs":"88abd5c16275bb2f2d77eaecf369d97681404a77b8edd0021f24bfd377c46be3","src/imp_cs.rs":"9eb73c340931f642664a8ee7a823af318c1118fab87b1aa63489e10a73c30945","src/imp_pl.rs":"9337fcbbfb80606de9f785f5e0f2d5ba513d7fd13918f90504bb24189831f877","src/imp_std.rs":"1c130f83be5c1360dfd379911f97797c1e4c730b845f465c8c2630467ca317d2","src/lib.rs":"9868277311543bf4a22463d41948af6e810feb40fdd7844d47893eeab1780215","src/race.rs":"4751464e8ccedb102097962a68d736de6f9434f78aa7761a1a03efd4a360c6c3","tests/it/main.rs":"e6e9987e053af84b9d76052602995b1e777efb5bc06cd5f49009e6f03b18626c","tests/it/race.rs":"5e299887123a852cb33692e27d88d59d2a007252675b60722d5e3ce5ca93dc19","tests/it/race_once_box.rs":"0cb5b3852f92002445ccc481de11642b22b9137f3a09db566cb484ab8eb32325","tests/it/sync_lazy.rs":"a36c5d66340b3d6d20aad331a499858a2125dfdfd624c5bf3b4b06a0b157c75c","tests/it/sync_once_cell.rs":"e6e539ce06966f656b24cc692603cf39241690fa58339d2775ecccda274b3769","tests/it/unsync_lazy.rs":"51a1ffd411770d1e32399ec23feb5f61be362bbed34e100eb7509f8496224e1a","tests/it/unsync_once_cell.rs":"82b72936d7bd4090db25cfc543c01ef3206d6917ac56f09d17d4110a65deb30a"},"package":"9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"}
\ No newline at end of file
### external/vendor/once_cell/.cargo_vcs_info.json
@@ -0,0 +1,6 @@
+{
+ "git": {
+ "sha1": "80fe900b21f6d76c1a2ed74d3343e8a3a88c46d0"
+ },
+ "path_in_vcs": ""
+}
\ No newline at end of file
### external/vendor/once_cell/.github/workflows/ci.yaml
@@ -0,0 +1,28 @@
+name: CI
+on:
+ pull_request:
+ push:
+ branches: ["master", "staging", "trying"]
+
+env:
+ CARGO_INCREMENTAL: 0
+ CARGO_NET_RETRY: 10
+ CI: 1
+ RUST_BACKTRACE: short
+ RUSTFLAGS: -D warnings
+ RUSTUP_MAX_RETRIES: 10
+
+jobs:
+ test:
+ name: Rust
+ runs-on: ubuntu-latest
+
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ fetch-depth: 0 # fetch tags for publish
+ - uses: Swatinem/rust-cache@359a70e43a0bb8a13953b04a90f76428b4959bb6
+ - run: cargo run -p xtask -- ci
+ env:
+ CARGO_REGISTRY_TOKEN: ${{ secrets.CRATES_IO_TOKEN }}
+ MIRIFLAGS: -Zmiri-strict-provenance
### external/vendor/once_cell/CHANGELOG.md
@@ -0,0 +1,263 @@
+# Changelog
+
+## 1.21.4
+
+- Fix unsoundness in `OnceCell::wait` under `--features=parking_lot`, [#295](https://github.com/matklad/once_cell/pull/295).
+
+ If thread A calls `wait`, while thread B calls `get_or_try_init(f)`, and, furthermore, `f` panics,
+ the `wait` incorrectly returns and thread A observes uninitialized memory.
+
+ Kudos to [@meng-xu-cs](https://github.com/meng-xu-cs) for a nice find!
+
+## 1.21.3
+
+- Outline more initialization in `race`: [#284](https://github.com/matklad/once_cell/pull/284),
+ [#285](https://github.com/matklad/once_cell/pull/285).
+
+## 1.21.2
+- Relax success ordering from AcqRel to Release in `race`: [#278](https://github.com/matklad/once_cell/pull/278).
+
+## 1.21.1
+- Reduce MSRV to 1.65: [#277](https://github.com/matklad/once_cell/pull/277).
+
+## 1.21.0
+
+- Outline initialization in `race`: [#273](https://github.com/matklad/once_cell/pull/273).
+- Add `OnceNonZereUsize::get_unchecked`: [#274](https://github.com/matklad/once_cell/pull/274).
+- Add `OnceBox::clone` and `OnceBox::with_value`: [#275](https://github.com/matklad/once_cell/pull/275).
+- Increase MSRV to 1.70
+
+## 1.20.2
+
+- Remove `portable_atomic` from Cargo.lock if it is not, in fact, used: [#267](https://github.com/matklad/once_cell/pull/267)
+ This is a work-around for this cargo bug: https://github.com/rust-lang/cargo/issues/10801.
+
+## 1.20.1
+
+- Allow using `race` module using just `portable_atomic`, without `critical_section` and provide
+ better error messages on targets without atomic CAS instruction,
+ [#265](https://github.com/matklad/once_cell/pull/265).
+
+## 1.19.0
+
+- Use `portable-atomic` instead of `atomic-polyfill`, [#251](https://github.com/matklad/once_cell/pull/251).
+
+## 1.18.0
+
+- `MSRV` is updated to 1.60.0 to take advantage of `dep:` syntax for cargo features,
+ removing "implementation details" from publicly visible surface.
+
+## 1.17.2
+
+- Avoid unnecessary synchronization in `Lazy::{force,deref}_mut()`, [#231](https://github.com/matklad/once_cell/pull/231).
+
+## 1.17.1
+
+- Make `OnceRef` implementation compliant with [strict provenance](https://github.com/rust-lang/rust/issues/95228).
+
+## 1.17.0
+
+- Add `race::OnceRef` for storing a `&'a T`.
+
+## 1.16.0
+
+- Add `no_std` implementation based on `critical-section`,
+ [#195](https://github.com/matklad/once_cell/pull/195).
+- Deprecate `atomic-polyfill` feature (use the new `critical-section` instead)
+
+## 1.15.0
+
+- Increase minimal supported Rust version to 1.56.0.
+- Implement `UnwindSafe` even if the `std` feature is disabled.
+
+## 1.14.0
+
+- Add extension to `unsync` and `sync` `Lazy` mut API:
+ - `force_mut`
+ - `get_mut`
+
+
+## 1.13.1
+
+- Make implementation compliant with [strict provenance](https://github.com/rust-lang/rust/issues/95228).
+- Upgrade `atomic-polyfill` to `1.0`
+
+## 1.13.0
+
+- Add `Lazy::get`, similar to `OnceCell::get`.
+
+## 1.12.1
+
+- Remove incorrect `debug_assert`.
+
+## 1.12.0
+
+- Add `OnceCell::wait`, a blocking variant of `get`.
+
+## 1.11.0
+
+- Add `OnceCell::with_value` to create initialized `OnceCell` in `const` context.
+- Improve `Clone` implementation for `OnceCell`.
+- Rewrite `parking_lot` version on top of `parking_lot_core`, for even smaller cells!
+
+## 1.10.0
+
+- upgrade `parking_lot` to `0.12.0` (note that this bumps MSRV with `parking_lot` feature enabled to `1.49.0`).
+
+## 1.9.0
+
+- Added an `atomic-polyfill` optional dependency to compile `race` on platforms without atomics
+
+## 1.8.0
+
+- Add `try_insert` API -- a version of `set` that returns a reference.
+
+## 1.7.2
+
+- Improve code size when using parking_lot feature.
+
+## 1.7.1
+
+- Fix `race::OnceBox<T>` to also impl `Default` even if `T` doesn't impl `Default`.
+
+## 1.7.0
+
+- Hide the `race` module behind (default) `race` feature.
+ Turns out that adding `race` by default was a breaking change on some platforms without atomics.
+ In this release, we make the module opt-out.
+ Technically, this is a breaking change for those who use `race` with `no_default_features`.
+ Given that the `race` module itself only several days old, the breakage is deemed acceptable.
+
+## 1.6.0
+
+- Add `Lazy::into_value`
+- Stabilize `once_cell::race` module for "first one wins" no_std-compatible initialization flavor.
+- Migrate from deprecated `compare_and_swap` to `compare_exchange`.
+
+## 1.5.2
+
+- `OnceBox` API uses `Box<T>`.
+ This a breaking change to unstable API.
+
+## 1.5.1
+
+- MSRV is increased to `1.36.0`.
+- document `once_cell::race` module.
+- introduce `alloc` feature for `OnceBox`.
+- fix `OnceBox::set`.
+
+## 1.5.0
+
+- add new `once_cell::race` module for "first one wins" no_std-compatible initialization flavor.
+ The API is provisional, subject to change and is gated by the `unstable` cargo feature.
+
+## 1.4.1
+
+- upgrade `parking_lot` to `0.11.0`
+- make `sync::OnceCell<T>` pass https://doc.rust-lang.org/nomicon/dropck.html#an-escape-hatch[dropck] with `parking_lot` feature enabled.
+ This fixes a (minor) semver-incompatible changed introduced in `1.4.0`
+
+## 1.4.0
+
+- upgrade `parking_lot` to `0.10` (note that this bumps MSRV with `parking_lot` feature enabled to `1.36.0`).
+- add `OnceCell::take`.
+- upgrade crossbeam utils (private dependency) to `0.7`.
+
+## 1.3.1
+
+- remove unnecessary `F: fmt::Debug` bound from `impl fmt::Debug for Lazy<T, F>`.
+
+## 1.3.0
+
+- `Lazy<T>` now implements `DerefMut`.
+- update implementation according to the latest changes in `std`.
+
+## 1.2.0
+
+- add `sync::OnceCell::get_unchecked`.
+
+## 1.1.0
+
+- implement `Default` for `Lazy`: it creates an empty `Lazy<T>` which is initialized with `T::default` on first access.
+- add `OnceCell::get_mut`.
+
+## 1.0.2
+
+- actually add `#![no_std]` attribute if std feature is not enabled.
+
+## 1.0.1
+
+- fix unsoundness in `Lazy<T>` if the initializing function panics. Thanks [@xfix](https://github.com/xfix)!
+- implement `RefUnwindSafe` for `Lazy`.
+- share more code between `std` and `parking_lot` implementations.
+- add F.A.Q section to the docs.
+
+## 1.0.0
+
+- remove `parking_lot` from the list of default features.
+- add `std` default feature. Without `std`, only `unsync` module is supported.
+- implement `Eq` for `OnceCell`.
+- fix wrong `Sync` bound on `sync::Lazy`.
+- run the whole test suite with miri.
+
+## 0.2.7
+
+- New implementation of `sync::OnceCell` if `parking_lot` feature is disabled.
+ It now employs a hand-rolled variant of `std::sync::Once`.
+- `sync::OnceCell::get_or_try_init` works without `parking_lot` as well!
+- document the effects of `parking_lot` feature: same performance but smaller types.
+
+## 0.2.6
+
+- Updated `Lazy`'s `Deref` impl to requires only `FnOnce` instead of `Fn`
+
+## 0.2.5
+
+- `Lazy` requires only `FnOnce` instead of `Fn`
+
+## 0.2.4
+
+- nicer `fmt::Debug` implementation
+
+## 0.2.3
+
+- update `parking_lot` to `0.9.0`
+- fix stacked borrows violation in `unsync::OnceCell::get`
+- implement `Clone` for `sync::OnceCell<T> where T: Clone`
+
+## 0.2.2
+
+- add `OnceCell::into_inner` which consumes a cell and returns an option
+
+## 0.2.1
+
+- implement `sync::OnceCell::get_or_try_init` if `parking_lot` feature is enabled
+- switch internal `unsafe` implementation of `sync::OnceCell` from `Once` to `Mutex`
+- `sync::OnceCell::get_or_init` is twice as fast if cell is already initialized
+- implement `std::panic::RefUnwindSafe` and `std::panic::UnwindSafe` for `OnceCell`
+- better document behavior around panics
+
+## 0.2.0
+
+- MSRV is now 1.31.1
+- `Lazy::new` and `OnceCell::new` are now const-fns
+- `unsync_lazy` and `sync_lazy` macros are removed
+
+## 0.1.8
+
+- update crossbeam-utils to 0.6
+- enable bors-ng
+
+## 0.1.7
+
+- cells implement `PartialEq` and `From`
+- MSRV is down to 1.24.1
+- update `parking_lot` to `0.7.1`
+
+## 0.1.6
+
+- `unsync::OnceCell<T>` is `Clone` if `T` is `Clone`.
+
+## 0.1.5
+
+- No changelog until this point :(
### external/vendor/once_cell/Cargo.lock
@@ -0,0 +1,179 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+version = 3
+
+[[package]]
+name = "aho-corasick"
+version = "1.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916"
+dependencies = [
+ "memchr",
+]
+
+[[package]]
+name = "bitflags"
+version = "2.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b048fb63fd8b5923fc5aa7b340d8e156aec7ec02f0c78fa8a6ddc2613f6f71de"
+
+[[package]]
+name = "cfg-if"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
+
+[[package]]
+name = "critical-section"
+version = "1.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f64009896348fc5af4222e9cf7d7d82a95a256c634ebcf61c53e4ea461422242"
+
+[[package]]
+name = "libc"
+version = "0.2.158"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d8adc4bb1803a324070e64a98ae98f38934d91957a99cfb3a43dcbc01bc56439"
+
+[[package]]
+name = "memchr"
+version = "2.7.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "78ca9ab1a0babb1e7d5695e3530886289c18cf2f87ec19a575a0abdce112e3a3"
+
+[[package]]
+name = "once_cell"
+version = "1.21.4"
+dependencies = [
+ "critical-section",
+ "parking_lot_core",
+ "portable-atomic",
+ "regex",
+]
+
+[[package]]
+name = "parking_lot_core"
+version = "0.9.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1e401f977ab385c9e4e3ab30627d6f26d00e2c73eef317493c4ec6d468726cf8"
+dependencies = [
+ "cfg-if",
+ "libc",
+ "redox_syscall",
+ "smallvec",
+ "windows-targets",
+]
+
+[[package]]
+name = "portable-atomic"
+version = "1.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cc9c68a3f6da06753e9335d63e27f6b9754dd1920d941135b7ea8224f141adb2"
+
+[[package]]
+name = "redox_syscall"
+version = "0.5.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0884ad60e090bf1345b93da0a5de8923c93884cd03f40dfcfddd3b4bee661853"
+dependencies = [
+ "bitflags",
+]
+
+[[package]]
+name = "regex"
+version = "1.10.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4219d74c6b67a3654a9fbebc4b419e22126d13d2f3c4a07ee0cb61ff79a79619"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-automata"
+version = "0.4.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "38caf58cc5ef2fed281f89292ef23f6365465ed9a41b7a7754eb4e26496c92df"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-syntax"
+version = "0.8.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7a66a03ae7c801facd77a29370b4faec201768915ac14a721ba36f20bc9c209b"
+
+[[package]]
+name = "smallvec"
+version = "1.13.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3c5e1a9a646d36c3599cd173a41282daf47c44583ad367b8e6837255952e5c67"
+
+[[package]]
+name = "windows-targets"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
+dependencies = [
+ "windows_aarch64_gnullvm",
+ "windows_aarch64_msvc",
+ "windows_i686_gnu",
+ "windows_i686_gnullvm",
+ "windows_i686_msvc",
+ "windows_x86_64_gnu",
+ "windows_x86_64_gnullvm",
+ "windows_x86_64_msvc",
+]
+
+[[package]]
+name = "windows_aarch64_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
+
+[[package]]
+name = "windows_aarch64_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
+
+[[package]]
+name = "windows_i686_gnu"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
+
+[[package]]
+name = "windows_i686_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
+
+[[package]]
+name = "windows_i686_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
+
+[[package]]
+name = "windows_x86_64_gnu"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
+
+[[package]]
+name = "windows_x86_64_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
+
+[[package]]
+name = "windows_x86_64_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
### external/vendor/once_cell/Cargo.toml
@@ -0,0 +1,119 @@
+# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
+#
+# When uploading crates to the registry Cargo will automatically
+# "normalize" Cargo.toml files for maximal compatibility
+# with all versions of Cargo and also rewrite `path` dependencies
+# to registry (e.g., crates.io) dependencies.
+#
+# If you are reading this file be aware that the original Cargo.toml
+# will likely look very different (and much more reasonable).
+# See Cargo.toml.orig for the original contents.
+
+[package]
+edition = "2021"
+rust-version = "1.65"
+name = "once_cell"
+version = "1.21.4"
+authors = ["Aleksey Kladov <aleksey.kladov@gmail.com>"]
+build = false
+exclude = [
+ "*.png",
+ "*.svg",
+ "/Cargo.lock.msrv",
+ "rustfmt.toml",
+]
+autolib = false
+autobins = false
+autoexamples = false
+autotests = false
+autobenches = false
+description = "Single assignment cells and lazy values."
+documentation = "https://docs.rs/once_cell"
+readme = "README.md"
+keywords = [
+ "lazy",
+ "static",
+]
+categories = [
+ "rust-patterns",
+ "memory-management",
+]
+license = "MIT OR Apache-2.0"
+repository = "https://github.com/matklad/once_cell"
+
+[package.metadata.docs.rs]
+all-features = true
+rustdoc-args = ["--generate-link-to-definition"]
+
+[features]
+alloc = ["race"]
+atomic-polyfill = ["critical-section"]
+critical-section = [
+ "dep:critical-section",
+ "portable-atomic",
+]
+default = ["std"]
+parking_lot = ["dep:parking_lot_core"]
+portable-atomic = ["dep:portable-atomic"]
+race = []
+std = ["alloc"]
+unstable = []
+
+[lib]
+name = "once_cell"
+path = "src/lib.rs"
+
+[[example]]
+name = "bench"
+path = "examples/bench.rs"
+required-features = ["std"]
+
+[[example]]
+name = "bench_acquire"
+path = "examples/bench_acquire.rs"
+required-features = ["std"]
+
+[[example]]
+name = "lazy_static"
+path = "examples/lazy_static.rs"
+required-features = ["std"]
+
+[[example]]
+name = "reentrant_init_deadlocks"
+path = "examples/reentrant_init_deadlocks.rs"
+required-features = ["std"]
+
+[[example]]
+name = "regex"
+path = "examples/regex.rs"
+required-features = ["std"]
+
+[[example]]
+name = "test_synchronization"
+path = "examples/test_synchronization.rs"
+required-features = ["std"]
+
+[[test]]
+name = "it"
+path = "tests/it/main.rs"
+
+[dependencies.critical-section]
+version = "1.1.3"
+optional = true
+
+[dependencies.parking_lot_core]
+version = "0.9.10"
+optional = true
+default-features = false
+
+[dependencies.portable-atomic]
+version = "1.8"
+optional = true
+default-features = false
+
+[dev-dependencies.critical-section]
+version = "1.1.3"
+features = ["std"]
+
+[dev-dependencies.regex]
+version = "1.10.6"
### external/vendor/once_cell/Cargo.toml.orig
@@ -0,0 +1,92 @@
+[package]
+name = "once_cell"
+version = "1.21.4"
+authors = ["Aleksey Kladov <aleksey.kladov@gmail.com>"]
+license = "MIT OR Apache-2.0"
+edition = "2021"
+rust-version = "1.65"
+
+description = "Single assignment cells and lazy values."
+readme = "README.md"
+documentation = "https://docs.rs/once_cell"
+
+repository = "https://github.com/matklad/once_cell"
+keywords = ["lazy", "static"]
+categories = ["rust-patterns", "memory-management"]
+
+exclude = ["*.png", "*.svg", "/Cargo.lock.msrv", "rustfmt.toml"]
+
+[workspace]
+members = ["xtask"]
+
+[dependencies]
+parking_lot_core = { version = "0.9.10", optional = true, default-features = false }
+portable-atomic = { version = "1.8", optional = true, default-features = false }
+critical-section = { version = "1.1.3", optional = true }
+
+[dev-dependencies]
+regex = "1.10.6"
+critical-section = { version = "1.1.3", features = ["std"] }
+
+[features]
+default = ["std"]
+
+# Enables `once_cell::sync` module.
+std = ["alloc"]
+
+# Enables `once_cell::race::OnceBox` type.
+alloc = ["race"]
+
+# Enables `once_cell::race` module.
+race = []
+
+# Uses parking_lot to implement once_cell::sync::OnceCell.
+# This makes no speed difference, but makes each OnceCell<T>
+# up to 16 bytes smaller, depending on the size of the T.
+parking_lot = ["dep:parking_lot_core"]
+
+# Uses `portable-atomic` to implement `race` module. in
+# `#![no_std]` mode. Please read `portable-atomic` docs carefully
+# before enabling this feature.
+portable-atomic = ["dep:portable-atomic"]
+
+# Uses `critical-section` to implement `sync` module. in
+# `#![no_std]` mode. Please read `critical-section` docs carefully
+# before enabling this feature.
+# `portable-atomic` feature is enabled for backwards compatibility.
+critical-section = ["dep:critical-section", "portable-atomic"]
+
+# Enables semver-exempt APIs of this crate.
+# At the moment, this feature is unused.
+unstable = []
+
+# Only for backwards compatibility.
+atomic-polyfill = ["critical-section"]
+
+[[example]]
+name = "bench"
+required-features = ["std"]
+
+[[example]]
+name = "bench_acquire"
+required-features = ["std"]
+
+[[example]]
+name = "lazy_static"
+required-features = ["std"]
+
+[[example]]
+name = "reentrant_init_deadlocks"
+required-features = ["std"]
+
+[[example]]
+name = "regex"
+required-features = ["std"]
+
+[[example]]
+name = "test_synchronization"
+required-features = ["std"]
+
+[package.metadata.docs.rs]
+all-features = true
+rustdoc-args = ["--generate-link-to-definition"]
### external/vendor/once_cell/LICENSE-APACHE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+Copyright [yyyy] [name of copyright owner]
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
### external/vendor/once_cell/LICENSE-MIT
@@ -0,0 +1,23 @@
+Permission is hereby granted, free of charge, to any
+person obtaining a copy of this software and associated
+documentation files (the "Software"), to deal in the
+Software without restriction, including without
+limitation the rights to use, copy, modify, merge,
+publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software
+is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice
+shall be included in all copies or substantial portions
+of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
+ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
+TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
+PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
+IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
### external/vendor/once_cell/README.md
@@ -0,0 +1,57 @@
+<p align="center"><img src="design/logo.png" alt="once_cell"></p>
+
+
+[](https://github.com/matklad/once_cell/actions)
+[](https://crates.io/crates/once_cell)
+[](https://docs.rs/once_cell/)
+
+# Overview
+
+`once_cell` provides two new cell-like types, `unsync::OnceCell` and `sync::OnceCell`. `OnceCell`
+might store arbitrary non-`Copy` types, can be assigned to at most once and provide direct access
+to the stored contents. In a nutshell, API looks *roughly* like this:
+
+```rust
+impl OnceCell<T> {
+ fn new() -> OnceCell<T> { ... }
+ fn set(&self, value: T) -> Result<(), T> { ... }
+ fn get(&self) -> Option<&T> { ... }
+}
+```
+
+Note that, like with `RefCell` and `Mutex`, the `set` method requires only a shared reference.
+Because of the single assignment restriction `get` can return an `&T` instead of `Ref<T>`
+or `MutexGuard<T>`.
+
+`once_cell` also has a `Lazy<T>` type, build on top of `OnceCell` which provides the same API as
+the `lazy_static!` macro, but without using any macros:
+
+```rust
+use std::{sync::Mutex, collections::HashMap};
+use once_cell::sync::Lazy;
+
+static GLOBAL_DATA: Lazy<Mutex<HashMap<i32, String>>> = Lazy::new(|| {
+ let mut m = HashMap::new();
+ m.insert(13, "Spica".to_string());
+ m.insert(74, "Hoyten".to_string());
+ Mutex::new(m)
+});
+
+fn main() {
+ println!("{:?}", GLOBAL_DATA.lock().unwrap());
+}
+```
+
+More patterns and use-cases are in the [docs](https://docs.rs/once_cell/)!
+
+# Related crates
+
+* [double-checked-cell](https://github.com/niklasf/double-checked-cell)
+* [lazy-init](https://crates.io/crates/lazy-init)
+* [lazycell](https://crates.io/crates/lazycell)
+* [mitochondria](https://crates.io/crates/mitochondria)
+* [lazy_static](https://crates.io/crates/lazy_static)
+* [async_once_cell](https://crates.io/crates/async_once_cell)
+* [generic_once_cell](https://crates.io/crates/generic_once_cell) (bring your own mutex)
+
+Parts of `once_cell` API are included into `std` [as of Rust 1.70.0](https://github.com/rust-lang/rust/pull/105587).
### external/vendor/once_cell/bors.toml
@@ -0,0 +1,2 @@
+status = [ "Rust" ]
+delete_merged_branches = true
### external/vendor/once_cell/examples/bench.rs
@@ -0,0 +1,28 @@
+use std::mem::size_of;
+
+use once_cell::sync::OnceCell;
+
+const N_THREADS: usize = 32;
+const N_ROUNDS: usize = 100_000_000;
+
+static CELL: OnceCell<usize> = OnceCell::new();
+
+fn main() {
+ let start = std::time::Instant::now();
+ let threads =
+ (0..N_THREADS).map(|i| std::thread::spawn(move || thread_main(i))).collect::<Vec<_>>();
+ for thread in threads {
+ thread.join().unwrap();
+ }
+ println!("{:?}", start.elapsed());
+ println!("size_of::<OnceCell<()>>() = {:?}", size_of::<OnceCell<()>>());
+ println!("size_of::<OnceCell<bool>>() = {:?}", size_of::<OnceCell<bool>>());
+ println!("size_of::<OnceCell<u32>>() = {:?}", size_of::<OnceCell<u32>>());
+}
+
+fn thread_main(i: usize) {
+ for _ in 0..N_ROUNDS {
+ let &value = CELL.get_or_init(|| i);
+ assert!(value < N_THREADS)
+ }
+}
### external/vendor/once_cell/examples/bench_acquire.rs
@@ -0,0 +1,39 @@
+//! Benchmark the overhead that the synchronization of `OnceCell::get` causes.
+//! We do some other operations that write to memory to get an imprecise but somewhat realistic
+//! measurement.
+
+use once_cell::sync::OnceCell;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+const N_THREADS: usize = 16;
+const N_ROUNDS: usize = 1_000_000;
+
+static CELL: OnceCell<usize> = OnceCell::new();
+static OTHER: AtomicUsize = AtomicUsize::new(0);
+
+fn main() {
+ let start = std::time::Instant::now();
+ let threads =
+ (0..N_THREADS).map(|i| std::thread::spawn(move || thread_main(i))).collect::<Vec<_>>();
+ for thread in threads {
+ thread.join().unwrap();
+ }
+ println!("{:?}", start.elapsed());
+ println!("{:?}", OTHER.load(Ordering::Relaxed));
+}
+
+#[inline(never)]
+fn thread_main(i: usize) {
+ // The operations we do here don't really matter, as long as we do multiple writes, and
+ // everything is messy enough to prevent the compiler from optimizing the loop away.
+ let mut data = [i; 128];
+ let mut accum = 0usize;
+ for _ in 0..N_ROUNDS {
+ let _value = CELL.get_or_init(|| i + 1);
+ let k = OTHER.fetch_add(data[accum & 0x7F] as usize, Ordering::Relaxed);
+ for j in data.iter_mut() {
+ *j = (*j).wrapping_add(accum);
+ accum = accum.wrapping_add(k);
+ }
+ }
+}
### external/vendor/once_cell/examples/lazy_static.rs
@@ -0,0 +1,36 @@
+extern crate once_cell;
+
+use once_cell::sync::{Lazy, OnceCell};
+use std::collections::HashMap;
+
+static HASHMAP: Lazy<HashMap<u32, &'static str>> = Lazy::new(|| {
+ let mut m = HashMap::new();
+ m.insert(0, "foo");
+ m.insert(1, "bar");
+ m.insert(2, "baz");
+ m
+});
+
+// Same, but completely without macros
+fn hashmap() -> &'static HashMap<u32, &'static str> {
+ static INSTANCE: OnceCell<HashMap<u32, &'static str>> = OnceCell::new();
+ INSTANCE.get_or_init(|| {
+ let mut m = HashMap::new();
+ m.insert(0, "foo");
+ m.insert(1, "bar");
+ m.insert(2, "baz");
+ m
+ })
+}
+
+fn main() {
+ // First access to `HASHMAP` initializes it
+ println!("The entry for `0` is \"{}\".", HASHMAP.get(&0).unwrap());
+
+ // Any further access to `HASHMAP` just returns the computed value
+ println!("The entry for `1` is \"{}\".", HASHMAP.get(&1).unwrap());
+
+ // The same works for function-style:
+ assert_eq!(hashmap().get(&0), Some(&"foo"));
+ assert_eq!(hashmap().get(&1), Some(&"bar"));
+}
### external/vendor/once_cell/examples/reentrant_init_deadlocks.rs
@@ -0,0 +1,14 @@
+fn main() {
+ let cell = once_cell::sync::OnceCell::<u32>::new();
+ cell.get_or_init(|| {
+ cell.get_or_init(|| 1);
+ 2
+ });
+}
+
+/// Dummy test to make it seem hang when compiled as `--test`
+/// See https://github.com/matklad/once_cell/issues/79
+#[test]
+fn dummy_test() {
+ std::thread::sleep(std::time::Duration::from_secs(4));
+}
### external/vendor/once_cell/examples/regex.rs
@@ -0,0 +1,49 @@
+use std::{str::FromStr, time::Instant};
+
+use regex::Regex;
+
+macro_rules! regex {
+ ($re:literal $(,)?) => {{
+ static RE: once_cell::sync::OnceCell<regex::Regex> = once_cell::sync::OnceCell::new();
+ RE.get_or_init(|| regex::Regex::new($re).unwrap())
+ }};
+}
+
+fn slow() {
+ let s = r##"13.28.24.13 - - [10/Mar/2016:19:29:25 +0100] "GET /etc/lib/pChart2/examples/index.php?Action=View&Script=../../../../cnf/db.php HTTP/1.1" 404 151 "-" "HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.16""##;
+
+ let mut total = 0;
+ for _ in 0..1000 {
+ let re = Regex::new(
+ r##"^(\S+) (\S+) (\S+) \[([^]]+)\] "([^"]*)" (\d+) (\d+) "([^"]*)" "([^"]*)"$"##,
+ )
+ .unwrap();
+ let size = usize::from_str(re.captures(s).unwrap().get(7).unwrap().as_str()).unwrap();
+ total += size;
+ }
+ println!("{}", total);
+}
+
+fn fast() {
+ let s = r##"13.28.24.13 - - [10/Mar/2016:19:29:25 +0100] "GET /etc/lib/pChart2/examples/index.php?Action=View&Script=../../../../cnf/db.php HTTP/1.1" 404 151 "-" "HTTP_Request2/2.2.1 (http://pear.php.net/package/http_request2) PHP/5.3.16""##;
+
+ let mut total = 0;
+ for _ in 0..1000 {
+ let re: &Regex = regex!(
+ r##"^(\S+) (\S+) (\S+) \[([^]]+)\] "([^"]*)" (\d+) (\d+) "([^"]*)" "([^"]*)"$"##,
+ );
+ let size = usize::from_str(re.captures(s).unwrap().get(7).unwrap().as_str()).unwrap();
+ total += size;
+ }
+ println!("{}", total);
+}
+
+fn main() {
+ let t = Instant::now();
+ slow();
+ println!("slow: {:?}", t.elapsed());
+
+ let t = Instant::now();
+ fast();
+ println!("fast: {:?}", t.elapsed());
+}
### external/vendor/once_cell/examples/test_synchronization.rs
@@ -0,0 +1,38 @@
+//! Test if the OnceCell properly synchronizes.
+//! Needs to be run in release mode.
+//!
+//! We create a `Vec` with `N_ROUNDS` of `OnceCell`s. All threads will walk the `Vec`, and race to
+//! be the first one to initialize a cell.
+//! Every thread adds the results of the cells it sees to an accumulator, which is compared at the
+//! end.
+//! All threads should end up with the same result.
+
+use once_cell::sync::OnceCell;
+
+const N_THREADS: usize = 32;
+const N_ROUNDS: usize = 1_000_000;
+
+static CELLS: OnceCell<Vec<OnceCell<usize>>> = OnceCell::new();
+static RESULT: OnceCell<usize> = OnceCell::new();
+
+fn main() {
+ let start = std::time::Instant::now();
+ CELLS.get_or_init(|| vec![OnceCell::new(); N_ROUNDS]);
+ let threads =
+ (0..N_THREADS).map(|i| std::thread::spawn(move || thread_main(i))).collect::<Vec<_>>();
+ for thread in threads {
+ thread.join().unwrap();
+ }
+ println!("{:?}", start.elapsed());
+ println!("No races detected");
+}
+
+fn thread_main(i: usize) {
+ let cells = CELLS.get().unwrap();
+ let mut accum = 0;
+ for cell in cells.iter() {
+ let &value = cell.get_or_init(|| i);
+ accum += value;
+ }
+ assert_eq!(RESULT.get_or_init(|| accum), &accum);
+}
### external/vendor/once_cell/src/imp_cs.rs
@@ -0,0 +1,78 @@
+use core::panic::{RefUnwindSafe, UnwindSafe};
+
+use critical_section::{CriticalSection, Mutex};
+use portable_atomic::{AtomicBool, Ordering};
+
+use crate::unsync;
+
+pub(crate) struct OnceCell<T> {
+ initialized: AtomicBool,
+ // Use `unsync::OnceCell` internally since `Mutex` does not provide
+ // interior mutability and to be able to re-use `get_or_try_init`.
+ value: Mutex<unsync::OnceCell<T>>,
+}
+
+// Why do we need `T: Send`?
+// Thread A creates a `OnceCell` and shares it with
+// scoped thread B, which fills the cell, which is
+// then destroyed by A. That is, destructor observes
+// a sent value.
+unsafe impl<T: Sync + Send> Sync for OnceCell<T> {}
+unsafe impl<T: Send> Send for OnceCell<T> {}
+
+impl<T: RefUnwindSafe + UnwindSafe> RefUnwindSafe for OnceCell<T> {}
+impl<T: UnwindSafe> UnwindSafe for OnceCell<T> {}
+
+impl<T> OnceCell<T> {
+ pub(crate) const fn new() -> OnceCell<T> {
+ OnceCell { initialized: AtomicBool::new(false), value: Mutex::new(unsync::OnceCell::new()) }
+ }
+
+ pub(crate) const fn with_value(value: T) -> OnceCell<T> {
+ OnceCell {
+ initialized: AtomicBool::new(true),
+ value: Mutex::new(unsync::OnceCell::with_value(value)),
+ }
+ }
+
+ #[inline]
+ pub(crate) fn is_initialized(&self) -> bool {
+ self.initialized.load(Ordering::Acquire)
+ }
+
+ #[cold]
+ pub(crate) fn initialize<F, E>(&self, f: F) -> Result<(), E>
+ where
+ F: FnOnce() -> Result<T, E>,
+ {
+ critical_section::with(|cs| {
+ let cell = self.value.borrow(cs);
+ cell.get_or_try_init(f).map(|_| {
+ self.initialized.store(true, Ordering::Release);
+ })
+ })
+ }
+
+ /// Get the reference to the underlying value, without checking if the cell
+ /// is initialized.
+ ///
+ /// # Safety
+ ///
+ /// Caller must ensure that the cell is in initialized state, and that
+ /// the contents are acquired by (synchronized to) this thread.
+ pub(crate) unsafe fn get_unchecked(&self) -> &T {
+ debug_assert!(self.is_initialized());
+ // SAFETY: The caller ensures that the value is initialized and access synchronized.
+ self.value.borrow(CriticalSection::new()).get().unwrap_unchecked()
+ }
+
+ #[inline]
+ pub(crate) fn get_mut(&mut self) -> Option<&mut T> {
+ self.value.get_mut().get_mut()
+ }
+
+ #[inline]
+ pub(crate) fn into_inner(self) -> Option<T> {
+ self.value.into_inner().into_inner()
+ }
+}
### external/vendor/once_cell/src/imp_pl.rs
@@ -0,0 +1,176 @@
+use std::{
+ cell::UnsafeCell,
+ panic::{RefUnwindSafe, UnwindSafe},
+ sync::atomic::{AtomicU8, Ordering},
+};
+
+pub(crate) struct OnceCell<T> {
+ state: AtomicU8,
+ value: UnsafeCell<Option<T>>,
+}
+
+const INCOMPLETE: u8 = 0x0;
+const RUNNING: u8 = 0x1;
+const COMPLETE: u8 = 0x2;
+
+// Why do we need `T: Send`?
+// Thread A creates a `OnceCell` and shares it with
+// scoped thread B, which fills the cell, which is
+// then destroyed by A. That is, destructor observes
+// a sent value.
+unsafe impl<T: Sync + Send> Sync for OnceCell<T> {}
+unsafe impl<T: Send> Send for OnceCell<T> {}
+
+impl<T: RefUnwindSafe + UnwindSafe> RefUnwindSafe for OnceCell<T> {}
+impl<T: UnwindSafe> UnwindSafe for OnceCell<T> {}
+
+impl<T> OnceCell<T> {
+ pub(crate) const fn new() -> OnceCell<T> {
+ OnceCell { state: AtomicU8::new(INCOMPLETE), value: UnsafeCell::new(None) }
+ }
+
+ pub(crate) const fn with_value(value: T) -> OnceCell<T> {
+ OnceCell { state: AtomicU8::new(COMPLETE), value: UnsafeCell::new(Some(value)) }
+ }
+
+ /// Safety: synchronizes with store to value via Release/Acquire.
+ #[inline]
+ pub(crate) fn is_initialized(&self) -> bool {
+ self.state.load(Ordering::Acquire) == COMPLETE
+ }
+
+ /// Safety: synchronizes with store to value via `is_initialized` or mutex
+ /// lock/unlock, writes value only once because of the mutex.
+ #[cold]
+ pub(crate) fn initialize<F, E>(&self, f: F) -> Result<(), E>
+ where
+ F: FnOnce() -> Result<T, E>,
+ {
+ let mut f = Some(f);
+ let mut res: Result<(), E> = Ok(());
+ let slot: *mut Option<T> = self.value.get();
+ initialize_inner(&self.state, &mut || {
+ // We are calling user-supplied function and need to be careful.
+ // - if it returns Err, we unlock mutex and return without touching anything
+ // - if it panics, we unlock mutex and propagate panic without touching anything
+ // - if it calls `set` or `get_or_try_init` re-entrantly, we get a deadlock on
+ // mutex, which is important for safety. We *could* detect this and panic,
+ // but that is more complicated
+ // - finally, if it returns Ok, we store the value and store the flag with
+ // `Release`, which synchronizes with `Acquire`s.
+ let f = unsafe { f.take().unwrap_unchecked() };
+ match f() {
+ Ok(value) => unsafe {
+ // Safe b/c we have a unique access and no panic may happen
+ // until the cell is marked as initialized.
+ debug_assert!((*slot).is_none());
+ *slot = Some(value);
+ true
+ },
+ Err(err) => {
+ res = Err(err);
+ false
+ }
+ }
+ });
+ res
+ }
+
+ #[cold]
+ pub(crate) fn wait(&self) {
+ let key = &self.state as *const _ as usize;
+ unsafe {
+ while self.state.load(Ordering::Acquire) != COMPLETE {
+ parking_lot_core::park(
+ key,
+ || self.state.load(Ordering::Acquire) != COMPLETE,
+ || (),
+ |_, _| (),
+ parking_lot_core::DEFAULT_PARK_TOKEN,
+ None,
+ );
+ }
+ }
+ }
+
+ /// Get the reference to the underlying value, without checking if the cell
+ /// is initialized.
+ ///
+ /// # Safety
+ ///
+ /// Caller must ensure that the cell is in initialized state, and that
+ /// the contents are acquired by (synchronized to) this thread.
+ pub(crate) unsafe fn get_unchecked(&self) -> &T {
+ debug_assert!(self.is_initialized());
+ let slot = &*self.value.get();
+ slot.as_ref().unwrap_unchecked()
+ }
+
+ /// Gets the mutable reference to the underlying value.
+ /// Returns `None` if the cell is empty.
+ pub(crate) fn get_mut(&mut self) -> Option<&mut T> {
+ // Safe b/c we have an exclusive access
+ let slot: &mut Option<T> = unsafe { &mut *self.value.get() };
+ slot.as_mut()
+ }
+
+ /// Consumes this `OnceCell`, returning the wrapped value.
+ /// Returns `None` if the cell was empty.
+ pub(crate) fn into_inner(self) -> Option<T> {
+ self.value.into_inner()
+ }
+}
+
+struct Guard<'a> {
+ state: &'a AtomicU8,
+ new_state: u8,
+}
+
+impl<'a> Drop for Guard<'a> {
+ fn drop(&mut self) {
+ self.state.store(self.new_state, Ordering::Release);
+ unsafe {
+ let key = self.state as *const AtomicU8 as usize;
+ parking_lot_core::unpark_all(key, parking_lot_core::DEFAULT_UNPARK_TOKEN);
+ }
+ }
+}
+
+// Note: this is intentionally monomorphic
+#[inline(never)]
+fn initialize_inner(state: &AtomicU8, init: &mut dyn FnMut() -> bool) {
+ loop {
+ let exchange =
+ state.compare_exchange_weak(INCOMPLETE, RUNNING, Ordering::Acquire, Ordering::Acquire);
+ match exchange {
+ Ok(_) => {
+ let mut guard = Guard { state, new_state: INCOMPLETE };
+ if init() {
+ guard.new_state = COMPLETE;
+ }
+ return;
+ }
+ Err(COMPLETE) => return,
+ Err(RUNNING) => unsafe {
+ let key = state as *const AtomicU8 as usize;
+ parking_lot_core::park(
+ key,
+ || state.load(Ordering::Relaxed) == RUNNING,
+ || (),
+ |_, _| (),
+ parking_lot_core::DEFAULT_PARK_TOKEN,
+ None,
+ );
+ },
+ Err(INCOMPLETE) => (),
+ Err(_) => debug_assert!(false),
+ }
+ }
+}
+
+#[test]
+fn test_size() {
+ use std::mem::size_of;
+
+ assert_eq!(size_of::<OnceCell<bool>>(), 1 * size_of::<bool>() + size_of::<u8>());
+}
### external/vendor/once_cell/src/imp_std.rs
@@ -0,0 +1,415 @@
+// There's a lot of scary concurrent code in this module, but it is copied from
+// `std::sync::Once` with two changes:
+// * no poisoning
+// * init function can fail
+
+use std::{
+ cell::{Cell, UnsafeCell},
+ panic::{RefUnwindSafe, UnwindSafe},
+ sync::atomic::{AtomicBool, AtomicPtr, Ordering},
+ thread::{self, Thread},
+};
+
+#[derive(Debug)]
+pub(crate) struct OnceCell<T> {
+ // This `queue` field is the core of the implementation. It encodes two
+ // pieces of information:
+ //
+ // * The current state of the cell (`INCOMPLETE`, `RUNNING`, `COMPLETE`)
+ // * Linked list of threads waiting for the current cell.
+ //
+ // State is encoded in two low bits. Only `INCOMPLETE` and `RUNNING` states
+ // allow waiters.
+ queue: AtomicPtr<Waiter>,
+ value: UnsafeCell<Option<T>>,
+}
+
+// Why do we need `T: Send`?
+// Thread A creates a `OnceCell` and shares it with
+// scoped thread B, which fills the cell, which is
+// then destroyed by A. That is, destructor observes
+// a sent value.
+unsafe impl<T: Sync + Send> Sync for OnceCell<T> {}
+unsafe impl<T: Send> Send for OnceCell<T> {}
+
+impl<T: RefUnwindSafe + UnwindSafe> RefUnwindSafe for OnceCell<T> {}
+impl<T: UnwindSafe> UnwindSafe for OnceCell<T> {}
+
+impl<T> OnceCell<T> {
+ pub(crate) const fn new() -> OnceCell<T> {
+ OnceCell { queue: AtomicPtr::new(INCOMPLETE_PTR), value: UnsafeCell::new(None) }
+ }
+
+ pub(crate) const fn with_value(value: T) -> OnceCell<T> {
+ OnceCell { queue: AtomicPtr::new(COMPLETE_PTR), value: UnsafeCell::new(Some(value)) }
+ }
+
+ /// Safety: synchronizes with store to value via Release/(Acquire|SeqCst).
+ #[inline]
+ pub(crate) fn is_initialized(&self) -> bool {
+ // An `Acquire` load is enough because that makes all the initialization
+ // operations visible to us, and, this being a fast path, weaker
+ // ordering helps with performance. This `Acquire` synchronizes with
+ // `SeqCst` operations on the slow path.
+ self.queue.load(Ordering::Acquire) == COMPLETE_PTR
+ }
+
+ /// Safety: synchronizes with store to value via SeqCst read from state,
+ /// writes value only once because we never get to INCOMPLETE state after a
+ /// successful write.
+ #[cold]
+ pub(crate) fn initialize<F, E>(&self, f: F) -> Result<(), E>
+ where
+ F: FnOnce() -> Result<T, E>,
+ {
+ let mut f = Some(f);
+ let mut res: Result<(), E> = Ok(());
+ let slot: *mut Option<T> = self.value.get();
+ initialize_or_wait(
+ &self.queue,
+ Some(&mut || {
+ let f = unsafe { f.take().unwrap_unchecked() };
+ match f() {
+ Ok(value) => {
+ unsafe { *slot = Some(value) };
+ true
+ }
+ Err(err) => {
+ res = Err(err);
+ false
+ }
+ }
+ }),
+ );
+ res
+ }
+
+ #[cold]
+ pub(crate) fn wait(&self) {
+ initialize_or_wait(&self.queue, None);
+ }
+
+ /// Get the reference to the underlying value, without checking if the cell
+ /// is initialized.
+ ///
+ /// # Safety
+ ///
+ /// Caller must ensure that the cell is in initialized state, and that
+ /// the contents are acquired by (synchronized to) this thread.
+ pub(crate) unsafe fn get_unchecked(&self) -> &T {
+ debug_assert!(self.is_initialized());
+ let slot = &*self.value.get();
+ slot.as_ref().unwrap_unchecked()
+ }
+
+ /// Gets the mutable reference to the underlying value.
+ /// Returns `None` if the cell is empty.
+ pub(crate) fn get_mut(&mut self) -> Option<&mut T> {
+ // Safe b/c we have a unique access.
+ unsafe { &mut *self.value.get() }.as_mut()
+ }
+
+ /// Consumes this `OnceCell`, returning the wrapped value.
+ /// Returns `None` if the cell was empty.
+ #[inline]
+ pub(crate) fn into_inner(self) -> Option<T> {
+ // Because `into_inner` takes `self` by value, the compiler statically
+ // verifies that it is not currently borrowed.
+ // So, it is safe to move out `Option<T>`.
+ self.value.into_inner()
+ }
+}
+
+// Three states that a OnceCell can be in, encoded into the lower bits of `queue` in
+// the OnceCell structure.
+const INCOMPLETE: usize = 0x0;
+const RUNNING: usize = 0x1;
+const COMPLETE: usize = 0x2;
+const INCOMPLETE_PTR: *mut Waiter = INCOMPLETE as *mut Waiter;
+const COMPLETE_PTR: *mut Waiter = COMPLETE as *mut Waiter;
+
+// Mask to learn about the state. All other bits are the queue of waiters if
+// this is in the RUNNING state.
+const STATE_MASK: usize = 0x3;
+
+/// Representation of a node in the linked list of waiters in the RUNNING state.
+/// A waiters is stored on the stack of the waiting threads.
+#[repr(align(4))] // Ensure the two lower bits are free to use as state bits.
+struct Waiter {
+ thread: Cell<Option<Thread>>,
+ signaled: AtomicBool,
+ next: *mut Waiter,
+}
+
+/// Drains and notifies the queue of waiters on drop.
+struct Guard<'a> {
+ queue: &'a AtomicPtr<Waiter>,
+ new_queue: *mut Waiter,
+}
+
+impl Drop for Guard<'_> {
+ fn drop(&mut self) {
+ let queue = self.queue.swap(self.new_queue, Ordering::AcqRel);
+
+ let state = strict::addr(queue) & STATE_MASK;
+ assert_eq!(state, RUNNING);
+
+ unsafe {
+ let mut waiter = strict::map_addr(queue, |q| q & !STATE_MASK);
+ while !waiter.is_null() {
+ let next = (*waiter).next;
+ let thread = (*waiter).thread.take().unwrap();
+ (*waiter).signaled.store(true, Ordering::Release);
+ waiter = next;
+ thread.unpark();
+ }
+ }
+ }
+}
+
+// Corresponds to `std::sync::Once::call_inner`.
+//
+// Originally copied from std, but since modified to remove poisoning and to
+// support wait.
+//
+// Note: this is intentionally monomorphic
+#[inline(never)]
+fn initialize_or_wait(queue: &AtomicPtr<Waiter>, mut init: Option<&mut dyn FnMut() -> bool>) {
+ let mut curr_queue = queue.load(Ordering::Acquire);
+
+ loop {
+ let curr_state = strict::addr(curr_queue) & STATE_MASK;
+ match (curr_state, &mut init) {
+ (COMPLETE, _) => return,
+ (INCOMPLETE, Some(init)) => {
+ let exchange = queue.compare_exchange(
+ curr_queue,
+ strict::map_addr(curr_queue, |q| (q & !STATE_MASK) | RUNNING),
+ Ordering::Acquire,
+ Ordering::Acquire,
+ );
+ if let Err(new_queue) = exchange {
+ curr_queue = new_queue;
+ continue;
+ }
+ let mut guard = Guard { queue, new_queue: INCOMPLETE_PTR };
+ if init() {
+ guard.new_queue = COMPLETE_PTR;
+ }
+ return;
+ }
+ (INCOMPLETE, None) | (RUNNING, _) => {
+ wait(queue, curr_queue);
+ curr_queue = queue.load(Ordering::Acquire);
+ }
+ _ => debug_assert!(false),
+ }
+ }
+}
+
+fn wait(queue: &AtomicPtr<Waiter>, mut curr_queue: *mut Waiter) {
+ let curr_state = strict::addr(curr_queue) & STATE_MASK;
+ loop {
+ let node = Waiter {
+ thread: Cell::new(Some(thread::current())),
+ signaled: AtomicBool::new(false),
+ next: strict::map_addr(curr_queue, |q| q & !STATE_MASK),
+ };
+ let me = &node as *const Waiter as *mut Waiter;
+
+ let exchange = queue.compare_exchange(
+ curr_queue,
+ strict::map_addr(me, |q| q | curr_state),
+ Ordering::Release,
+ Ordering::Relaxed,
+ );
+ if let Err(new_queue) = exchange {
+ if strict::addr(new_queue) & STATE_MASK != curr_state {
+ return;
+ }
+ curr_queue = new_queue;
+ continue;
+ }
+
+ while !node.signaled.load(Ordering::Acquire) {
+ thread::park();
+ }
+ break;
+ }
+}
+
+// Polyfill of strict provenance from https://crates.io/crates/sptr.
+//
+// Use free-standing function rather than a trait to keep things simple and
+// avoid any potential conflicts with future stabile std API.
+mod strict {
+ #[must_use]
+ #[inline]
+ pub(crate) fn addr<T>(ptr: *mut T) -> usize
+ where
+ T: Sized,
+ {
+ // FIXME(strict_provenance_magic): I am magic and should be a compiler intrinsic.
+ // SAFETY: Pointer-to-integer transmutes are valid (if you are okay with losing the
+ // provenance).
+ unsafe { core::mem::transmute(ptr) }
+ }
+
+ #[must_use]
+ #[inline]
+ pub(crate) fn with_addr<T>(ptr: *mut T, addr: usize) -> *mut T
+ where
+ T: Sized,
+ {
+ // FIXME(strict_provenance_magic): I am magic and should be a compiler intrinsic.
+ //
+ // In the mean-time, this operation is defined to be "as if" it was
+ // a wrapping_offset, so we can emulate it as such. This should properly
+ // restore pointer provenance even under today's compiler.
+ let self_addr = self::addr(ptr) as isize;
+ let dest_addr = addr as isize;
+ let offset = dest_addr.wrapping_sub(self_addr);
+
+ // This is the canonical desugarring of this operation,
+ // but `pointer::cast` was only stabilized in 1.38.
+ // self.cast::<u8>().wrapping_offset(offset).cast::<T>()
+ (ptr as *mut u8).wrapping_offset(offset) as *mut T
+ }
+
+ #[must_use]
+ #[inline]
+ pub(crate) fn map_addr<T>(ptr: *mut T, f: impl FnOnce(usize) -> usize) -> *mut T
+ where
+ T: Sized,
+ {
+ self::with_addr(ptr, f(addr(ptr)))
+ }
+}
+
+// These test are snatched from std as well.
+#[cfg(test)]
+mod tests {
+ use std::panic;
+ use std::{sync::mpsc::channel, thread};
+
+ use super::OnceCell;
+
+ impl<T> OnceCell<T> {
+ fn init(&self, f: impl FnOnce() -> T) {
+ enum Void {}
+ let _ = self.initialize(|| Ok::<T, Void>(f()));
+ }
+ }
+
+ #[test]
+ fn smoke_once() {
+ static O: OnceCell<()> = OnceCell::new();
+ let mut a = 0;
+ O.init(|| a += 1);
+ assert_eq!(a, 1);
+ O.init(|| a += 1);
+ assert_eq!(a, 1);
+ }
+
+ #[test]
+ fn stampede_once() {
+ static O: OnceCell<()> = OnceCell::new();
+ static mut RUN: bool = false;
+
+ let (tx, rx) = channel();
+ for _ in 0..10 {
+ let tx = tx.clone();
+ thread::spawn(move || {
+ for _ in 0..4 {
+ thread::yield_now()
+ }
+ unsafe {
+ O.init(|| {
+ assert!(!RUN);
+ RUN = true;
+ });
+ assert!(RUN);
+ }
+ tx.send(()).unwrap();
+ });
+ }
+
+ unsafe {
+ O.init(|| {
+ assert!(!RUN);
+ RUN = true;
+ });
+ assert!(RUN);
+ }
+
+ for _ in 0..10 {
+ rx.recv().unwrap();
+ }
+ }
+
+ #[test]
+ fn poison_bad() {
+ static O: OnceCell<()> = OnceCell::new();
+
+ // poison the once
+ let t = panic::catch_unwind(|| {
+ O.init(|| panic!());
+ });
+ assert!(t.is_err());
+
+ // we can subvert poisoning, however
+ let mut called = false;
+ O.init(|| {
+ called = true;
+ });
+ assert!(called);
+
+ // once any success happens, we stop propagating the poison
+ O.init(|| {});
+ }
+
+ #[test]
+ fn wait_for_force_to_finish() {
+ static O: OnceCell<()> = OnceCell::new();
+
+ // poison the once
+ let t = panic::catch_unwind(|| {
+ O.init(|| panic!());
+ });
+ assert!(t.is_err());
+
+ // make sure someone's waiting inside the once via a force
+ let (tx1, rx1) = channel();
+ let (tx2, rx2) = channel();
+ let t1 = thread::spawn(move || {
+ O.init(|| {
+ tx1.send(()).unwrap();
+ rx2.recv().unwrap();
+ });
+ });
+
+ rx1.recv().unwrap();
+
+ // put another waiter on the once
+ let t2 = thread::spawn(|| {
+ let mut called = false;
+ O.init(|| {
+ called = true;
+ });
+ assert!(!called);
+ });
+
+ tx2.send(()).unwrap();
+
+ assert!(t1.join().is_ok());
+ assert!(t2.join().is_ok());
+ }
+
+ #[test]
+ #[cfg(target_pointer_width = "64")]
+ fn test_size() {
+ use std::mem::size_of;
+
+ assert_eq!(size_of::<OnceCell<u32>>(), 4 * size_of::<u32>());
+ }
+}
### external/vendor/once_cell/src/lib.rs
@@ -0,0 +1,1422 @@
+//! # Overview
+//!
+//! `once_cell` provides two new cell-like types, [`unsync::OnceCell`] and
+//! [`sync::OnceCell`]. A `OnceCell` might store arbitrary non-`Copy` types, can
+//! be assigned to at most once and provides direct access to the stored
+//! contents. The core API looks *roughly* like this (and there's much more
+//! inside, read on!):
+//!
+//! ```rust,ignore
+//! impl<T> OnceCell<T> {
+//! const fn new() -> OnceCell<T> { ... }
+//! fn set(&self, value: T) -> Result<(), T> { ... }
+//! fn get(&self) -> Option<&T> { ... }
+//! }
+//! ```
+//!
+//! Note that, like with [`RefCell`] and [`Mutex`], the `set` method requires
+//! only a shared reference. Because of the single assignment restriction `get`
+//! can return a `&T` instead of `Ref<T>` or `MutexGuard<T>`.
+//!
+//! The `sync` flavor is thread-safe (that is, implements the [`Sync`] trait),
+//! while the `unsync` one is not.
+//!
+//! [`unsync::OnceCell`]: unsync/struct.OnceCell.html
+//! [`sync::OnceCell`]: sync/struct.OnceCell.html
+//! [`RefCell`]: https://doc.rust-lang.org/std/cell/struct.RefCell.html
+//! [`Mutex`]: https://doc.rust-lang.org/std/sync/struct.Mutex.html
+//! [`Sync`]: https://doc.rust-lang.org/std/marker/trait.Sync.html
+//!
+//! # Recipes
+//!
+//! `OnceCell` might be useful for a variety of patterns.
+//!
+//! ## Safe Initialization of Global Data
+//!
+//! ```rust
+//! # #[cfg(any(feature = "std", feature = "critical-section"))] {
+//! use std::{env, io};
+//!
+//! use once_cell::sync::OnceCell;
+//!
+//! #[derive(Debug)]
+//! pub struct Logger {
+//! // ...
+//! }
+//! static INSTANCE: OnceCell<Logger> = OnceCell::new();
+//!
+//! impl Logger {
+//! pub fn global() -> &'static Logger {
+//! INSTANCE.get().expect("logger is not initialized")
+//! }
+//!
+//! fn from_cli(args: env::Args) -> Result<Logger, std::io::Error> {
+//! // ...
+//! # Ok(Logger {})
+//! }
+//! }
+//!
+//! fn main() {
+//! let logger = Logger::from_cli(env::args()).unwrap();
+//! INSTANCE.set(logger).unwrap();
+//! // use `Logger::global()` from now on
+//! }
+//! # }
+//! ```
+//!
+//! ## Lazy Initialized Global Data
+//!
+//! This is essentially the `lazy_static!` macro, but without a macro.
+//!
+//! ```rust
+//! # #[cfg(any(feature = "std", feature = "critical-section"))] {
+//! use std::{sync::Mutex, collections::HashMap};
+//!
+//! use once_cell::sync::OnceCell;
+//!
+//! fn global_data() -> &'static Mutex<HashMap<i32, String>> {
+//! static INSTANCE: OnceCell<Mutex<HashMap<i32, String>>> = OnceCell::new();
+//! INSTANCE.get_or_init(|| {
+//! let mut m = HashMap::new();
+//! m.insert(13, "Spica".to_string());
+//! m.insert(74, "Hoyten".to_string());
+//! Mutex::new(m)
+//! })
+//! }
+//! # }
+//! ```
+//!
+//! There are also the [`sync::Lazy`] and [`unsync::Lazy`] convenience types to
+//! streamline this pattern:
+//!
+//! ```rust
+//! # #[cfg(any(feature = "std", feature = "critical-section"))] {
+//! use std::{sync::Mutex, collections::HashMap};
+//! use once_cell::sync::Lazy;
+//!
+//! static GLOBAL_DATA: Lazy<Mutex<HashMap<i32, String>>> = Lazy::new(|| {
+//! let mut m = HashMap::new();
+//! m.insert(13, "Spica".to_string());
+//! m.insert(74, "Hoyten".to_string());
+//! Mutex::new(m)
+//! });
+//!
+//! fn main() {
+//! println!("{:?}", GLOBAL_DATA.lock().unwrap());
+//! }
+//! # }
+//! ```
+//!
+//! Note that the variable that holds `Lazy` is declared as `static`, *not*
+//! `const`. This is important: using `const` instead compiles, but works wrong.
+//!
+//! [`sync::Lazy`]: sync/struct.Lazy.html
+//! [`unsync::Lazy`]: unsync/struct.Lazy.html
+//!
+//! ## General purpose lazy evaluation
+//!
+//! Unlike `lazy_static!`, `Lazy` works with local variables.
+//!
+//! ```rust
+//! use once_cell::unsync::Lazy;
+//!
+//! fn main() {
+//! let ctx = vec![1, 2, 3];
+//! let thunk = Lazy::new(|| {
+//! ctx.iter().sum::<i32>()
+//! });
+//! assert_eq!(*thunk, 6);
+//! }
+//! ```
+//!
+//! If you need a lazy field in a struct, you probably should use `OnceCell`
+//! directly, because that will allow you to access `self` during
+//! initialization.
+//!
+//! ```rust
+//! use std::{fs, path::PathBuf};
+//!
+//! use once_cell::unsync::OnceCell;
+//!
+//! struct Ctx {
+//! config_path: PathBuf,
+//! config: OnceCell<String>,
+//! }
+//!
+//! impl Ctx {
+//! pub fn get_config(&self) -> Result<&str, std::io::Error> {
+//! let cfg = self.config.get_or_try_init(|| {
+//! fs::read_to_string(&self.config_path)
+//! })?;
+//! Ok(cfg.as_str())
+//! }
+//! }
+//! ```
+//!
+//! ## Lazily Compiled Regex
+//!
+//! This is a `regex!` macro which takes a string literal and returns an
+//! *expression* that evaluates to a `&'static Regex`:
+//!
+//! ```
+//! macro_rules! regex {
+//! ($re:literal $(,)?) => {{
+//! static RE: once_cell::sync::OnceCell<regex::Regex> = once_cell::sync::OnceCell::new();
+//! RE.get_or_init(|| regex::Regex::new($re).unwrap())
+//! }};
+//! }
+//! ```
+//!
+//! This macro can be useful to avoid the "compile regex on every loop
+//! iteration" problem.
+//!
+//! ## Runtime `include_bytes!`
+//!
+//! The `include_bytes` macro is useful to include test resources, but it slows
+//! down test compilation a lot. An alternative is to load the resources at
+//! runtime:
+//!
+//! ```
+//! # #[cfg(any(feature = "std", feature = "critical-section"))] {
+//! use std::path::Path;
+//!
+//! use once_cell::sync::OnceCell;
+//!
+//! pub struct TestResource {
+//! path: &'static str,
+//! cell: OnceCell<Vec<u8>>,
+//! }
+//!
+//! impl TestResource {
+//! pub const fn new(path: &'static str) -> TestResource {
+//! TestResource { path, cell: OnceCell::new() }
+//! }
+//! pub fn bytes(&self) -> &[u8] {
+//! self.cell.get_or_init(|| {
+//! let dir = std::env::var("CARGO_MANIFEST_DIR").unwrap();
+//! let path = Path::new(dir.as_str()).join(self.path);
+//! std::fs::read(&path).unwrap_or_else(|_err| {
+//! panic!("failed to load test resource: {}", path.display())
+//! })
+//! }).as_slice()
+//! }
+//! }
+//!
+//! static TEST_IMAGE: TestResource = TestResource::new("test_data/lena.png");
+//!
+//! #[test]
+//! fn test_sobel_filter() {
+//! let rgb: &[u8] = TEST_IMAGE.bytes();
+//! // ...
+//! # drop(rgb);
+//! }
+//! # }
+//! ```
+//!
+//! ## `lateinit`
+//!
+//! `LateInit` type for delayed initialization. It is reminiscent of Kotlin's
+//! `lateinit` keyword and allows construction of cyclic data structures:
+//!
+//!
+//! ```
+//! # #[cfg(any(feature = "std", feature = "critical-section"))] {
+//! use once_cell::sync::OnceCell;
+//!
+//! pub struct LateInit<T> { cell: OnceCell<T> }
+//!
+//! impl<T> LateInit<T> {
+//! pub fn init(&self, value: T) {
+//! assert!(self.cell.set(value).is_ok())
+//! }
+//! }
+//!
+//! impl<T> Default for LateInit<T> {
+//! fn default() -> Self { LateInit { cell: OnceCell::default() } }
+//! }
+//!
+//! impl<T> std::ops::Deref for LateInit<T> {
+//! type Target = T;
+//! fn deref(&self) -> &T {
+//! self.cell.get().unwrap()
+//! }
+//! }
+//!
+//! #[derive(Default)]
+//! struct A<'a> {
+//! b: LateInit<&'a B<'a>>,
+//! }
+//!
+//! #[derive(Default)]
+//! struct B<'a> {
+//! a: LateInit<&'a A<'a>>
+//! }
+//!
+//!
+//! fn build_cycle() {
+//! let a = A::default();
+//! let b = B::default();
+//! a.b.init(&b);
+//! b.a.init(&a);
+//!
+//! let _a = &a.b.a.b.a;
+//! }
+//! # }
+//! ```
+//!
+//! # Comparison with std
+//!
+//! |`!Sync` types | Access Mode | Drawbacks |
+//! |----------------------|------------------------|-----------------------------------------------|
+//! |`Cell<T>` | `T` | requires `T: Copy` for `get` |
+//! |`RefCell<T>` | `RefMut<T>` / `Ref<T>` | may panic at runtime |
+//! |`unsync::OnceCell<T>` | `&T` | assignable only once |
+//!
+//! |`Sync` types | Access Mode | Drawbacks |
+//! |----------------------|------------------------|-----------------------------------------------|
+//! |`AtomicT` | `T` | works only with certain `Copy` types |
+//! |`Mutex<T>` | `MutexGuard<T>` | may deadlock at runtime, may block the thread |
+//! |`sync::OnceCell<T>` | `&T` | assignable only once, may block the thread |
+//!
+//! Technically, calling `get_or_init` will also cause a panic or a deadlock if
+//! it recursively calls itself. However, because the assignment can happen only
+//! once, such cases should be more rare than equivalents with `RefCell` and
+//! `Mutex`.
+//!
+//! # Minimum Supported `rustc` Version
+//!
+//! If only the `std`, `alloc`, or `race` features are enabled, MSRV will be
+//! updated conservatively, supporting at least latest 8 versions of the compiler.
+//! When using other features, like `parking_lot`, MSRV might be updated more
+//! frequently, up to the latest stable. In both cases, increasing MSRV is *not*
+//! considered a semver-breaking change and requires only a minor version bump.
+//!
+//! # Implementation details
+//!
+//! The implementation is based on the
+//! [`lazy_static`](https://github.com/rust-lang-nursery/lazy-static.rs/) and
+//! [`lazy_cell`](https://github.com/indiv0/lazycell/) crates and
+//! [`std::sync::Once`]. In some sense, `once_cell` just streamlines and unifies
+//! those APIs.
+//!
+//! To implement a sync flavor of `OnceCell`, this crates uses either a custom
+//! re-implementation of `std::sync::Once` or `parking_lot::Mutex`. This is
+//! controlled by the `parking_lot` feature (disabled by default). Performance
+//! is the same for both cases, but the `parking_lot` based `OnceCell<T>` is
+//! smaller by up to 16 bytes.
+//!
+//! This crate uses `unsafe`.
+//!
+//! [`std::sync::Once`]: https://doc.rust-lang.org/std/sync/struct.Once.html
+//!
+//! # F.A.Q.
+//!
+//! **Should I use the sync or unsync flavor?**
+//!
+//! Because Rust compiler checks thread safety for you, it's impossible to
+//! accidentally use `unsync` where `sync` is required. So, use `unsync` in
+//! single-threaded code and `sync` in multi-threaded. It's easy to switch
+//! between the two if code becomes multi-threaded later.
+//!
+//! At the moment, `unsync` has an additional benefit that reentrant
+//! initialization causes a panic, which might be easier to debug than a
+//! deadlock.
+//!
+//! **Does this crate support async?**
+//!
+//! No, but you can use
+//! [`async_once_cell`](https://crates.io/crates/async_once_cell) instead.
+//!
+//! **Does this crate support `no_std`?**
+//!
+//! Yes, but with caveats. `OnceCell` is a synchronization primitive which
+//! _semantically_ relies on blocking. `OnceCell` guarantees that at most one
+//! `f` will be called to compute the value. If two threads of execution call
+//! `get_or_init` concurrently, one of them has to wait.
+//!
+//! Waiting fundamentally requires OS support. Execution environment needs to
+//! understand who waits on whom to prevent deadlocks due to priority inversion.
+//! You _could_ make code to compile by blindly using pure spinlocks, but the
+//! runtime behavior would be subtly wrong.
+//!
+//! Given these constraints, `once_cell` provides the following options:
+//!
+//! - The `race` module provides similar, but distinct synchronization primitive
+//! which is compatible with `no_std`. With `race`, the `f` function can be
+//! called multiple times by different threads, but only one thread will win
+//! to install the value.
+//! - `critical-section` feature (with a `-`, not `_`) uses `critical_section`
+//! to implement blocking.
+//!
+//! **Can I bring my own mutex?**
+//!
+//! There is [generic_once_cell](https://crates.io/crates/generic_once_cell) to
+//! allow just that.
+//!
+//! **Should I use `std::cell::OnceCell`, `once_cell`, or `lazy_static`?**
+//!
+//! If you can use `std` version (your MSRV is at least 1.70, and you don't need
+//! extra features `once_cell` provides), use `std`. Otherwise, use `once_cell`.
+//! Don't use `lazy_static`.
+//!
+//! # Related crates
+//!
+//! * Most of this crate's functionality is available in `std` starting with
+//! Rust 1.70. See `std::cell::OnceCell` and `std::sync::OnceLock`.
+//! * [double-checked-cell](https://github.com/niklasf/double-checked-cell)
+//! * [lazy-init](https://crates.io/crates/lazy-init)
+//! * [lazycell](https://crates.io/crates/lazycell)
+//! * [mitochondria](https://crates.io/crates/mitochondria)
+//! * [lazy_static](https://crates.io/crates/lazy_static)
+//! * [async_once_cell](https://crates.io/crates/async_once_cell)
+//! * [generic_once_cell](https://crates.io/crates/generic_once_cell) (bring
+//! your own mutex)
+
+#![cfg_attr(not(feature = "std"), no_std)]
+
+#[cfg(feature = "alloc")]
+extern crate alloc;
+
+#[cfg(all(feature = "critical-section", not(feature = "std")))]
+#[path = "imp_cs.rs"]
+mod imp;
+
+#[cfg(all(feature = "std", feature = "parking_lot"))]
+#[path = "imp_pl.rs"]
+mod imp;
+
+#[cfg(all(feature = "std", not(feature = "parking_lot")))]
+#[path = "imp_std.rs"]
+mod imp;
+
+/// Single-threaded version of `OnceCell`.
+pub mod unsync {
+ use core::{
+ cell::{Cell, UnsafeCell},
+ fmt, mem,
+ ops::{Deref, DerefMut},
+ panic::{RefUnwindSafe, UnwindSafe},
+ };
+
+ /// A cell which can be written to only once. It is not thread safe.
+ ///
+ /// Unlike [`std::cell::RefCell`], a `OnceCell` provides simple `&`
+ /// references to the contents.
+ ///
+ /// [`std::cell::RefCell`]: https://doc.rust-lang.org/std/cell/struct.RefCell.html
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::OnceCell;
+ ///
+ /// let cell = OnceCell::new();
+ /// assert!(cell.get().is_none());
+ ///
+ /// let value: &String = cell.get_or_init(|| {
+ /// "Hello, World!".to_string()
+ /// });
+ /// assert_eq!(value, "Hello, World!");
+ /// assert!(cell.get().is_some());
+ /// ```
+ pub struct OnceCell<T> {
+ // Invariant: written to at most once.
+ inner: UnsafeCell<Option<T>>,
+ }
+
+ // Similarly to a `Sync` bound on `sync::OnceCell`, we can use
+ // `&unsync::OnceCell` to sneak a `T` through `catch_unwind`,
+ // by initializing the cell in closure and extracting the value in the
+ // `Drop`.
+ impl<T: RefUnwindSafe + UnwindSafe> RefUnwindSafe for OnceCell<T> {}
+ impl<T: UnwindSafe> UnwindSafe for OnceCell<T> {}
+
+ impl<T> Default for OnceCell<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+ }
+
+ impl<T: fmt::Debug> fmt::Debug for OnceCell<T> {
+ fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+ match self.get() {
+ Some(v) => f.debug_tuple("OnceCell").field(v).finish(),
+ None => f.write_str("OnceCell(Uninit)"),
+ }
+ }
+ }
+
+ impl<T: Clone> Clone for OnceCell<T> {
+ fn clone(&self) -> OnceCell<T> {
+ match self.get() {
+ Some(value) => OnceCell::with_value(value.clone()),
+ None => OnceCell::new(),
+ }
+ }
+
+ fn clone_from(&mut self, source: &Self) {
+ match (self.get_mut(), source.get()) {
+ (Some(this), Some(source)) => this.clone_from(source),
+ _ => *self = source.clone(),
+ }
+ }
+ }
+
+ impl<T: PartialEq> PartialEq for OnceCell<T> {
+ fn eq(&self, other: &Self) -> bool {
+ self.get() == other.get()
+ }
+ }
+
+ impl<T: Eq> Eq for OnceCell<T> {}
+
+ impl<T> From<T> for OnceCell<T> {
+ fn from(value: T) -> Self {
+ OnceCell::with_value(value)
+ }
+ }
+
+ impl<T> OnceCell<T> {
+ /// Creates a new empty cell.
+ pub const fn new() -> OnceCell<T> {
+ OnceCell { inner: UnsafeCell::new(None) }
+ }
+
+ /// Creates a new initialized cell.
+ pub const fn with_value(value: T) -> OnceCell<T> {
+ OnceCell { inner: UnsafeCell::new(Some(value)) }
+ }
+
+ /// Gets a reference to the underlying value.
+ ///
+ /// Returns `None` if the cell is empty.
+ #[inline]
+ pub fn get(&self) -> Option<&T> {
+ // Safe due to `inner`'s invariant of being written to at most once.
+ // Had multiple writes to `inner` been allowed, a reference to the
+ // value we return now would become dangling by a write of a
+ // different value later.
+ unsafe { &*self.inner.get() }.as_ref()
+ }
+
+ /// Gets a mutable reference to the underlying value.
+ ///
+ /// Returns `None` if the cell is empty.
+ ///
+ /// This method is allowed to violate the invariant of writing to a `OnceCell`
+ /// at most once because it requires `&mut` access to `self`. As with all
+ /// interior mutability, `&mut` access permits arbitrary modification:
+ ///
+ /// ```
+ /// use once_cell::unsync::OnceCell;
+ ///
+ /// let mut cell: OnceCell<u32> = OnceCell::new();
+ /// cell.set(92).unwrap();
+ /// *cell.get_mut().unwrap() = 93;
+ /// assert_eq!(cell.get(), Some(&93));
+ /// ```
+ #[inline]
+ pub fn get_mut(&mut self) -> Option<&mut T> {
+ // Safe because we have unique access
+ unsafe { &mut *self.inner.get() }.as_mut()
+ }
+
+ /// Sets the contents of this cell to `value`.
+ ///
+ /// Returns `Ok(())` if the cell was empty and `Err(value)` if it was
+ /// full.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::OnceCell;
+ ///
+ /// let cell = OnceCell::new();
+ /// assert!(cell.get().is_none());
+ ///
+ /// assert_eq!(cell.set(92), Ok(()));
+ /// assert_eq!(cell.set(62), Err(62));
+ ///
+ /// assert!(cell.get().is_some());
+ /// ```
+ pub fn set(&self, value: T) -> Result<(), T> {
+ match self.try_insert(value) {
+ Ok(_) => Ok(()),
+ Err((_, value)) => Err(value),
+ }
+ }
+
+ /// Like [`set`](Self::set), but also returns a reference to the final cell value.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::OnceCell;
+ ///
+ /// let cell = OnceCell::new();
+ /// assert!(cell.get().is_none());
+ ///
+ /// assert_eq!(cell.try_insert(92), Ok(&92));
+ /// assert_eq!(cell.try_insert(62), Err((&92, 62)));
+ ///
+ /// assert!(cell.get().is_some());
+ /// ```
+ pub fn try_insert(&self, value: T) -> Result<&T, (&T, T)> {
+ if let Some(old) = self.get() {
+ return Err((old, value));
+ }
+
+ let slot = unsafe { &mut *self.inner.get() };
+ // This is the only place where we set the slot, no races
+ // due to reentrancy/concurrency are possible, and we've
+ // checked that slot is currently `None`, so this write
+ // maintains the `inner`'s invariant.
+ *slot = Some(value);
+ Ok(unsafe { slot.as_ref().unwrap_unchecked() })
+ }
+
+ /// Gets the contents of the cell, initializing it with `f`
+ /// if the cell was empty.
+ ///
+ /// # Panics
+ ///
+ /// If `f` panics, the panic is propagated to the caller, and the cell
+ /// remains uninitialized.
+ ///
+ /// It is an error to reentrantly initialize the cell from `f`. Doing
+ /// so results in a panic.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::OnceCell;
+ ///
+ /// let cell = OnceCell::new();
+ /// let value = cell.get_or_init(|| 92);
+ /// assert_eq!(value, &92);
+ /// let value = cell.get_or_init(|| unreachable!());
+ /// assert_eq!(value, &92);
+ /// ```
+ pub fn get_or_init<F>(&self, f: F) -> &T
+ where
+ F: FnOnce() -> T,
+ {
+ enum Void {}
+ match self.get_or_try_init(|| Ok::<T, Void>(f())) {
+ Ok(val) => val,
+ Err(void) => match void {},
+ }
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if
+ /// the cell was empty. If the cell was empty and `f` failed, an
+ /// error is returned.
+ ///
+ /// # Panics
+ ///
+ /// If `f` panics, the panic is propagated to the caller, and the cell
+ /// remains uninitialized.
+ ///
+ /// It is an error to reentrantly initialize the cell from `f`. Doing
+ /// so results in a panic.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::OnceCell;
+ ///
+ /// let cell = OnceCell::new();
+ /// assert_eq!(cell.get_or_try_init(|| Err(())), Err(()));
+ /// assert!(cell.get().is_none());
+ /// let value = cell.get_or_try_init(|| -> Result<i32, ()> {
+ /// Ok(92)
+ /// });
+ /// assert_eq!(value, Ok(&92));
+ /// assert_eq!(cell.get(), Some(&92))
+ /// ```
+ pub fn get_or_try_init<F, E>(&self, f: F) -> Result<&T, E>
+ where
+ F: FnOnce() -> Result<T, E>,
+ {
+ if let Some(val) = self.get() {
+ return Ok(val);
+ }
+ let val = f()?;
+ // Note that *some* forms of reentrant initialization might lead to
+ // UB (see `reentrant_init` test). I believe that just removing this
+ // `assert`, while keeping `set/get` would be sound, but it seems
+ // better to panic, rather than to silently use an old value.
+ assert!(self.set(val).is_ok(), "reentrant init");
+ Ok(unsafe { self.get().unwrap_unchecked() })
+ }
+
+ /// Takes the value out of this `OnceCell`, moving it back to an uninitialized state.
+ ///
+ /// Has no effect and returns `None` if the `OnceCell` hasn't been initialized.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use once_cell::unsync::OnceCell;
+ ///
+ /// let mut cell: OnceCell<String> = OnceCell::new();
+ /// assert_eq!(cell.take(), None);
+ ///
+ /// let mut cell = OnceCell::new();
+ /// cell.set("hello".to_string()).unwrap();
+ /// assert_eq!(cell.take(), Some("hello".to_string()));
+ /// assert_eq!(cell.get(), None);
+ /// ```
+ ///
+ /// This method is allowed to violate the invariant of writing to a `OnceCell`
+ /// at most once because it requires `&mut` access to `self`. As with all
+ /// interior mutability, `&mut` access permits arbitrary modification:
+ ///
+ /// ```
+ /// use once_cell::unsync::OnceCell;
+ ///
+ /// let mut cell: OnceCell<u32> = OnceCell::new();
+ /// cell.set(92).unwrap();
+ /// cell = OnceCell::new();
+ /// ```
+ pub fn take(&mut self) -> Option<T> {
+ mem::take(self).into_inner()
+ }
+
+ /// Consumes the `OnceCell`, returning the wrapped value.
+ ///
+ /// Returns `None` if the cell was empty.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use once_cell::unsync::OnceCell;
+ ///
+ /// let cell: OnceCell<String> = OnceCell::new();
+ /// assert_eq!(cell.into_inner(), None);
+ ///
+ /// let cell = OnceCell::new();
+ /// cell.set("hello".to_string()).unwrap();
+ /// assert_eq!(cell.into_inner(), Some("hello".to_string()));
+ /// ```
+ pub fn into_inner(self) -> Option<T> {
+ // Because `into_inner` takes `self` by value, the compiler statically verifies
+ // that it is not currently borrowed. So it is safe to move out `Option<T>`.
+ self.inner.into_inner()
+ }
+ }
+
+ /// A value which is initialized on the first access.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::Lazy;
+ ///
+ /// let lazy: Lazy<i32> = Lazy::new(|| {
+ /// println!("initializing");
+ /// 92
+ /// });
+ /// println!("ready");
+ /// println!("{}", *lazy);
+ /// println!("{}", *lazy);
+ ///
+ /// // Prints:
+ /// // ready
+ /// // initializing
+ /// // 92
+ /// // 92
+ /// ```
+ pub struct Lazy<T, F = fn() -> T> {
+ cell: OnceCell<T>,
+ init: Cell<Option<F>>,
+ }
+
+ impl<T, F: RefUnwindSafe> RefUnwindSafe for Lazy<T, F> where OnceCell<T>: RefUnwindSafe {}
+
+ impl<T: fmt::Debug, F> fmt::Debug for Lazy<T, F> {
+ fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+ f.debug_struct("Lazy").field("cell", &self.cell).field("init", &"..").finish()
+ }
+ }
+
+ impl<T, F> Lazy<T, F> {
+ /// Creates a new lazy value with the given initializing function.
+ ///
+ /// # Example
+ /// ```
+ /// # fn main() {
+ /// use once_cell::unsync::Lazy;
+ ///
+ /// let hello = "Hello, World!".to_string();
+ ///
+ /// let lazy = Lazy::new(|| hello.to_uppercase());
+ ///
+ /// assert_eq!(&*lazy, "HELLO, WORLD!");
+ /// # }
+ /// ```
+ pub const fn new(init: F) -> Lazy<T, F> {
+ Lazy { cell: OnceCell::new(), init: Cell::new(Some(init)) }
+ }
+
+ /// Consumes this `Lazy` returning the stored value.
+ ///
+ /// Returns `Ok(value)` if `Lazy` is initialized and `Err(f)` otherwise.
+ pub fn into_value(this: Lazy<T, F>) -> Result<T, F> {
+ let cell = this.cell;
+ let init = this.init;
+ cell.into_inner().ok_or_else(|| {
+ init.take().unwrap_or_else(|| panic!("Lazy instance has previously been poisoned"))
+ })
+ }
+ }
+
+ impl<T, F: FnOnce() -> T> Lazy<T, F> {
+ /// Forces the evaluation of this lazy value and returns a reference to
+ /// the result.
+ ///
+ /// This is equivalent to the `Deref` impl, but is explicit.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::Lazy;
+ ///
+ /// let lazy = Lazy::new(|| 92);
+ ///
+ /// assert_eq!(Lazy::force(&lazy), &92);
+ /// assert_eq!(&*lazy, &92);
+ /// ```
+ pub fn force(this: &Lazy<T, F>) -> &T {
+ this.cell.get_or_init(|| match this.init.take() {
+ Some(f) => f(),
+ None => panic!("Lazy instance has previously been poisoned"),
+ })
+ }
+
+ /// Forces the evaluation of this lazy value and returns a mutable reference to
+ /// the result.
+ ///
+ /// This is equivalent to the `DerefMut` impl, but is explicit.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::Lazy;
+ ///
+ /// let mut lazy = Lazy::new(|| 92);
+ ///
+ /// assert_eq!(Lazy::force_mut(&mut lazy), &92);
+ /// assert_eq!(*lazy, 92);
+ /// ```
+ pub fn force_mut(this: &mut Lazy<T, F>) -> &mut T {
+ if this.cell.get_mut().is_none() {
+ let value = match this.init.get_mut().take() {
+ Some(f) => f(),
+ None => panic!("Lazy instance has previously been poisoned"),
+ };
+ this.cell = OnceCell::with_value(value);
+ }
+ this.cell.get_mut().unwrap_or_else(|| unreachable!())
+ }
+
+ /// Gets the reference to the result of this lazy value if
+ /// it was initialized, otherwise returns `None`.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::Lazy;
+ ///
+ /// let lazy = Lazy::new(|| 92);
+ ///
+ /// assert_eq!(Lazy::get(&lazy), None);
+ /// assert_eq!(&*lazy, &92);
+ /// assert_eq!(Lazy::get(&lazy), Some(&92));
+ /// ```
+ pub fn get(this: &Lazy<T, F>) -> Option<&T> {
+ this.cell.get()
+ }
+
+ /// Gets the mutable reference to the result of this lazy value if
+ /// it was initialized, otherwise returns `None`.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::unsync::Lazy;
+ ///
+ /// let mut lazy = Lazy::new(|| 92);
+ ///
+ /// assert_eq!(Lazy::get_mut(&mut lazy), None);
+ /// assert_eq!(*lazy, 92);
+ /// assert_eq!(Lazy::get_mut(&mut lazy), Some(&mut 92));
+ /// ```
+ pub fn get_mut(this: &mut Lazy<T, F>) -> Option<&mut T> {
+ this.cell.get_mut()
+ }
+ }
+
+ impl<T, F: FnOnce() -> T> Deref for Lazy<T, F> {
+ type Target = T;
+ fn deref(&self) -> &T {
+ Lazy::force(self)
+ }
+ }
+
+ impl<T, F: FnOnce() -> T> DerefMut for Lazy<T, F> {
+ fn deref_mut(&mut self) -> &mut T {
+ Lazy::force_mut(self)
+ }
+ }
+
+ impl<T: Default> Default for Lazy<T> {
+ /// Creates a new lazy value using `Default` as the initializing function.
+ fn default() -> Lazy<T> {
+ Lazy::new(T::default)
+ }
+ }
+}
+
+/// Thread-safe, blocking version of `OnceCell`.
+#[cfg(any(feature = "std", feature = "critical-section"))]
+pub mod sync {
+ use core::{
+ cell::Cell,
+ fmt, mem,
+ ops::{Deref, DerefMut},
+ panic::RefUnwindSafe,
+ };
+
+ use super::imp::OnceCell as Imp;
+
+ /// A thread-safe cell which can be written to only once.
+ ///
+ /// `OnceCell` provides `&` references to the contents without RAII guards.
+ ///
+ /// Reading a non-`None` value out of `OnceCell` establishes a
+ /// happens-before relationship with a corresponding write. For example, if
+ /// thread A initializes the cell with `get_or_init(f)`, and thread B
+ /// subsequently reads the result of this call, B also observes all the side
+ /// effects of `f`.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// static CELL: OnceCell<String> = OnceCell::new();
+ /// assert!(CELL.get().is_none());
+ ///
+ /// std::thread::spawn(|| {
+ /// let value: &String = CELL.get_or_init(|| {
+ /// "Hello, World!".to_string()
+ /// });
+ /// assert_eq!(value, "Hello, World!");
+ /// }).join().unwrap();
+ ///
+ /// let value: Option<&String> = CELL.get();
+ /// assert!(value.is_some());
+ /// assert_eq!(value.unwrap().as_str(), "Hello, World!");
+ /// ```
+ pub struct OnceCell<T>(Imp<T>);
+
+ impl<T> Default for OnceCell<T> {
+ fn default() -> OnceCell<T> {
+ OnceCell::new()
+ }
+ }
+
+ impl<T: fmt::Debug> fmt::Debug for OnceCell<T> {
+ fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+ match self.get() {
+ Some(v) => f.debug_tuple("OnceCell").field(v).finish(),
+ None => f.write_str("OnceCell(Uninit)"),
+ }
+ }
+ }
+
+ impl<T: Clone> Clone for OnceCell<T> {
+ fn clone(&self) -> OnceCell<T> {
+ match self.get() {
+ Some(value) => Self::with_value(value.clone()),
+ None => Self::new(),
+ }
+ }
+
+ fn clone_from(&mut self, source: &Self) {
+ match (self.get_mut(), source.get()) {
+ (Some(this), Some(source)) => this.clone_from(source),
+ _ => *self = source.clone(),
+ }
+ }
+ }
+
+ impl<T> From<T> for OnceCell<T> {
+ fn from(value: T) -> Self {
+ Self::with_value(value)
+ }
+ }
+
+ impl<T: PartialEq> PartialEq for OnceCell<T> {
+ fn eq(&self, other: &OnceCell<T>) -> bool {
+ self.get() == other.get()
+ }
+ }
+
+ impl<T: Eq> Eq for OnceCell<T> {}
+
+ impl<T> OnceCell<T> {
+ /// Creates a new empty cell.
+ pub const fn new() -> OnceCell<T> {
+ OnceCell(Imp::new())
+ }
+
+ /// Creates a new initialized cell.
+ pub const fn with_value(value: T) -> OnceCell<T> {
+ OnceCell(Imp::with_value(value))
+ }
+
+ /// Gets the reference to the underlying value.
+ ///
+ /// Returns `None` if the cell is empty, or being initialized. This
+ /// method never blocks.
+ pub fn get(&self) -> Option<&T> {
+ if self.0.is_initialized() {
+ // Safe b/c value is initialized.
+ Some(unsafe { self.get_unchecked() })
+ } else {
+ None
+ }
+ }
+
+ /// Gets the reference to the underlying value, blocking the current
+ /// thread until it is set.
+ ///
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// let mut cell = std::sync::Arc::new(OnceCell::new());
+ /// let t = std::thread::spawn({
+ /// let cell = std::sync::Arc::clone(&cell);
+ /// move || cell.set(92).unwrap()
+ /// });
+ ///
+ /// // Returns immediately, but might return None.
+ /// let _value_or_none = cell.get();
+ ///
+ /// // Will return 92, but might block until the other thread does `.set`.
+ /// let value: &u32 = cell.wait();
+ /// assert_eq!(*value, 92);
+ /// t.join().unwrap();
+ /// ```
+ #[cfg(feature = "std")]
+ pub fn wait(&self) -> &T {
+ if !self.0.is_initialized() {
+ self.0.wait()
+ }
+ debug_assert!(self.0.is_initialized());
+ // Safe b/c of the wait call above and the fact that we didn't
+ // relinquish our borrow.
+ unsafe { self.get_unchecked() }
+ }
+
+ /// Gets the mutable reference to the underlying value.
+ ///
+ /// Returns `None` if the cell is empty.
+ ///
+ /// This method is allowed to violate the invariant of writing to a `OnceCell`
+ /// at most once because it requires `&mut` access to `self`. As with all
+ /// interior mutability, `&mut` access permits arbitrary modification:
+ ///
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// let mut cell: OnceCell<u32> = OnceCell::new();
+ /// cell.set(92).unwrap();
+ /// cell = OnceCell::new();
+ /// ```
+ #[inline]
+ pub fn get_mut(&mut self) -> Option<&mut T> {
+ self.0.get_mut()
+ }
+
+ /// Get the reference to the underlying value, without checking if the
+ /// cell is initialized.
+ ///
+ /// # Safety
+ ///
+ /// Caller must ensure that the cell is in initialized state, and that
+ /// the contents are acquired by (synchronized to) this thread.
+ #[inline]
+ pub unsafe fn get_unchecked(&self) -> &T {
+ self.0.get_unchecked()
+ }
+
+ /// Sets the contents of this cell to `value`.
+ ///
+ /// Returns `Ok(())` if the cell was empty and `Err(value)` if it was
+ /// full.
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// static CELL: OnceCell<i32> = OnceCell::new();
+ ///
+ /// fn main() {
+ /// assert!(CELL.get().is_none());
+ ///
+ /// std::thread::spawn(|| {
+ /// assert_eq!(CELL.set(92), Ok(()));
+ /// }).join().unwrap();
+ ///
+ /// assert_eq!(CELL.set(62), Err(62));
+ /// assert_eq!(CELL.get(), Some(&92));
+ /// }
+ /// ```
+ pub fn set(&self, value: T) -> Result<(), T> {
+ match self.try_insert(value) {
+ Ok(_) => Ok(()),
+ Err((_, value)) => Err(value),
+ }
+ }
+
+ /// Like [`set`](Self::set), but also returns a reference to the final cell value.
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// let cell = OnceCell::new();
+ /// assert!(cell.get().is_none());
+ ///
+ /// assert_eq!(cell.try_insert(92), Ok(&92));
+ /// assert_eq!(cell.try_insert(62), Err((&92, 62)));
+ ///
+ /// assert!(cell.get().is_some());
+ /// ```
+ pub fn try_insert(&self, value: T) -> Result<&T, (&T, T)> {
+ let mut value = Some(value);
+ let res = self.get_or_init(|| unsafe { value.take().unwrap_unchecked() });
+ match value {
+ None => Ok(res),
+ Some(value) => Err((res, value)),
+ }
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if the cell
+ /// was empty.
+ ///
+ /// Many threads may call `get_or_init` concurrently with different
+ /// initializing functions, but it is guaranteed that only one function
+ /// will be executed.
+ ///
+ /// # Panics
+ ///
+ /// If `f` panics, the panic is propagated to the caller, and the cell
+ /// remains uninitialized.
+ ///
+ /// It is an error to reentrantly initialize the cell from `f`. The
+ /// exact outcome is unspecified. Current implementation deadlocks, but
+ /// this may be changed to a panic in the future.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// let cell = OnceCell::new();
+ /// let value = cell.get_or_init(|| 92);
+ /// assert_eq!(value, &92);
+ /// let value = cell.get_or_init(|| unreachable!());
+ /// assert_eq!(value, &92);
+ /// ```
+ pub fn get_or_init<F>(&self, f: F) -> &T
+ where
+ F: FnOnce() -> T,
+ {
+ enum Void {}
+ match self.get_or_try_init(|| Ok::<T, Void>(f())) {
+ Ok(val) => val,
+ Err(void) => match void {},
+ }
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if
+ /// the cell was empty. If the cell was empty and `f` failed, an
+ /// error is returned.
+ ///
+ /// # Panics
+ ///
+ /// If `f` panics, the panic is propagated to the caller, and
+ /// the cell remains uninitialized.
+ ///
+ /// It is an error to reentrantly initialize the cell from `f`.
+ /// The exact outcome is unspecified. Current implementation
+ /// deadlocks, but this may be changed to a panic in the future.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// let cell = OnceCell::new();
+ /// assert_eq!(cell.get_or_try_init(|| Err(())), Err(()));
+ /// assert!(cell.get().is_none());
+ /// let value = cell.get_or_try_init(|| -> Result<i32, ()> {
+ /// Ok(92)
+ /// });
+ /// assert_eq!(value, Ok(&92));
+ /// assert_eq!(cell.get(), Some(&92))
+ /// ```
+ pub fn get_or_try_init<F, E>(&self, f: F) -> Result<&T, E>
+ where
+ F: FnOnce() -> Result<T, E>,
+ {
+ // Fast path check
+ if let Some(value) = self.get() {
+ return Ok(value);
+ }
+
+ self.0.initialize(f)?;
+
+ // Safe b/c value is initialized.
+ debug_assert!(self.0.is_initialized());
+ Ok(unsafe { self.get_unchecked() })
+ }
+
+ /// Takes the value out of this `OnceCell`, moving it back to an uninitialized state.
+ ///
+ /// Has no effect and returns `None` if the `OnceCell` hasn't been initialized.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// let mut cell: OnceCell<String> = OnceCell::new();
+ /// assert_eq!(cell.take(), None);
+ ///
+ /// let mut cell = OnceCell::new();
+ /// cell.set("hello".to_string()).unwrap();
+ /// assert_eq!(cell.take(), Some("hello".to_string()));
+ /// assert_eq!(cell.get(), None);
+ /// ```
+ ///
+ /// This method is allowed to violate the invariant of writing to a `OnceCell`
+ /// at most once because it requires `&mut` access to `self`. As with all
+ /// interior mutability, `&mut` access permits arbitrary modification:
+ ///
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// let mut cell: OnceCell<u32> = OnceCell::new();
+ /// cell.set(92).unwrap();
+ /// cell = OnceCell::new();
+ /// ```
+ pub fn take(&mut self) -> Option<T> {
+ mem::take(self).into_inner()
+ }
+
+ /// Consumes the `OnceCell`, returning the wrapped value. Returns
+ /// `None` if the cell was empty.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use once_cell::sync::OnceCell;
+ ///
+ /// let cell: OnceCell<String> = OnceCell::new();
+ /// assert_eq!(cell.into_inner(), None);
+ ///
+ /// let cell = OnceCell::new();
+ /// cell.set("hello".to_string()).unwrap();
+ /// assert_eq!(cell.into_inner(), Some("hello".to_string()));
+ /// ```
+ #[inline]
+ pub fn into_inner(self) -> Option<T> {
+ self.0.into_inner()
+ }
+ }
+
+ /// A value which is initialized on the first access.
+ ///
+ /// This type is thread-safe and can be used in statics.
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use std::collections::HashMap;
+ ///
+ /// use once_cell::sync::Lazy;
+ ///
+ /// static HASHMAP: Lazy<HashMap<i32, String>> = Lazy::new(|| {
+ /// println!("initializing");
+ /// let mut m = HashMap::new();
+ /// m.insert(13, "Spica".to_string());
+ /// m.insert(74, "Hoyten".to_string());
+ /// m
+ /// });
+ ///
+ /// fn main() {
+ /// println!("ready");
+ /// std::thread::spawn(|| {
+ /// println!("{:?}", HASHMAP.get(&13));
+ /// }).join().unwrap();
+ /// println!("{:?}", HASHMAP.get(&74));
+ ///
+ /// // Prints:
+ /// // ready
+ /// // initializing
+ /// // Some("Spica")
+ /// // Some("Hoyten")
+ /// }
+ /// ```
+ pub struct Lazy<T, F = fn() -> T> {
+ cell: OnceCell<T>,
+ init: Cell<Option<F>>,
+ }
+
+ impl<T: fmt::Debug, F> fmt::Debug for Lazy<T, F> {
+ fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+ f.debug_struct("Lazy").field("cell", &self.cell).field("init", &"..").finish()
+ }
+ }
+
+ // We never create a `&F` from a `&Lazy<T, F>` so it is fine to not impl
+ // `Sync` for `F`. We do create a `&mut Option<F>` in `force`, but this is
+ // properly synchronized, so it only happens once so it also does not
+ // contribute to this impl.
+ unsafe impl<T, F: Send> Sync for Lazy<T, F> where OnceCell<T>: Sync {}
+ // auto-derived `Send` impl is OK.
+
+ impl<T, F: RefUnwindSafe> RefUnwindSafe for Lazy<T, F> where OnceCell<T>: RefUnwindSafe {}
+
+ impl<T, F> Lazy<T, F> {
+ /// Creates a new lazy value with the given initializing
+ /// function.
+ pub const fn new(f: F) -> Lazy<T, F> {
+ Lazy { cell: OnceCell::new(), init: Cell::new(Some(f)) }
+ }
+
+ /// Consumes this `Lazy` returning the stored value.
+ ///
+ /// Returns `Ok(value)` if `Lazy` is initialized and `Err(f)` otherwise.
+ pub fn into_value(this: Lazy<T, F>) -> Result<T, F> {
+ let cell = this.cell;
+ let init = this.init;
+ cell.into_inner().ok_or_else(|| {
+ init.take().unwrap_or_else(|| panic!("Lazy instance has previously been poisoned"))
+ })
+ }
+ }
+
+ impl<T, F: FnOnce() -> T> Lazy<T, F> {
+ /// Forces the evaluation of this lazy value and
+ /// returns a reference to the result. This is equivalent
+ /// to the `Deref` impl, but is explicit.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::sync::Lazy;
+ ///
+ /// let lazy = Lazy::new(|| 92);
+ ///
+ /// assert_eq!(Lazy::force(&lazy), &92);
+ /// assert_eq!(&*lazy, &92);
+ /// ```
+ pub fn force(this: &Lazy<T, F>) -> &T {
+ this.cell.get_or_init(|| match this.init.take() {
+ Some(f) => f(),
+ None => panic!("Lazy instance has previously been poisoned"),
+ })
+ }
+
+ /// Forces the evaluation of this lazy value and
+ /// returns a mutable reference to the result. This is equivalent
+ /// to the `Deref` impl, but is explicit.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::sync::Lazy;
+ ///
+ /// let mut lazy = Lazy::new(|| 92);
+ ///
+ /// assert_eq!(Lazy::force_mut(&mut lazy), &mut 92);
+ /// ```
+ pub fn force_mut(this: &mut Lazy<T, F>) -> &mut T {
+ if this.cell.get_mut().is_none() {
+ let value = match this.init.get_mut().take() {
+ Some(f) => f(),
+ None => panic!("Lazy instance has previously been poisoned"),
+ };
+ this.cell = OnceCell::with_value(value);
+ }
+ this.cell.get_mut().unwrap_or_else(|| unreachable!())
+ }
+
+ /// Gets the reference to the result of this lazy value if
+ /// it was initialized, otherwise returns `None`.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::sync::Lazy;
+ ///
+ /// let lazy = Lazy::new(|| 92);
+ ///
+ /// assert_eq!(Lazy::get(&lazy), None);
+ /// assert_eq!(&*lazy, &92);
+ /// assert_eq!(Lazy::get(&lazy), Some(&92));
+ /// ```
+ pub fn get(this: &Lazy<T, F>) -> Option<&T> {
+ this.cell.get()
+ }
+
+ /// Gets the reference to the result of this lazy value if
+ /// it was initialized, otherwise returns `None`.
+ ///
+ /// # Example
+ /// ```
+ /// use once_cell::sync::Lazy;
+ ///
+ /// let mut lazy = Lazy::new(|| 92);
+ ///
+ /// assert_eq!(Lazy::get_mut(&mut lazy), None);
+ /// assert_eq!(&*lazy, &92);
+ /// assert_eq!(Lazy::get_mut(&mut lazy), Some(&mut 92));
+ /// ```
+ pub fn get_mut(this: &mut Lazy<T, F>) -> Option<&mut T> {
+ this.cell.get_mut()
+ }
+ }
+
+ impl<T, F: FnOnce() -> T> Deref for Lazy<T, F> {
+ type Target = T;
+ fn deref(&self) -> &T {
+ Lazy::force(self)
+ }
+ }
+
+ impl<T, F: FnOnce() -> T> DerefMut for Lazy<T, F> {
+ fn deref_mut(&mut self) -> &mut T {
+ Lazy::force_mut(self)
+ }
+ }
+
+ impl<T: Default> Default for Lazy<T> {
+ /// Creates a new lazy value using `Default` as the initializing function.
+ fn default() -> Lazy<T> {
+ Lazy::new(T::default)
+ }
+ }
+
+ /// ```compile_fail
+ /// struct S(*mut ());
+ /// unsafe impl Sync for S {}
+ ///
+ /// fn share<T: Sync>(_: &T) {}
+ /// share(&once_cell::sync::OnceCell::<S>::new());
+ /// ```
+ ///
+ /// ```compile_fail
+ /// struct S(*mut ());
+ /// unsafe impl Sync for S {}
+ ///
+ /// fn share<T: Sync>(_: &T) {}
+ /// share(&once_cell::sync::Lazy::<S>::new(|| unimplemented!()));
+ /// ```
+ fn _dummy() {}
+}
+
+#[cfg(feature = "race")]
+pub mod race;
### external/vendor/once_cell/src/race.rs
@@ -0,0 +1,498 @@
+//! Thread-safe, non-blocking, "first one wins" flavor of `OnceCell`.
+//!
+//! If two threads race to initialize a type from the `race` module, they
+//! don't block, execute initialization function together, but only one of
+//! them stores the result.
+//!
+//! This module does not require `std` feature.
+//!
+//! # Atomic orderings
+//!
+//! All types in this module use `Acquire` and `Release`
+//! [atomic orderings](Ordering) for all their operations. While this is not
+//! strictly necessary for types other than `OnceBox`, it is useful for users as
+//! it allows them to be certain that after `get` or `get_or_init` returns on
+//! one thread, any side-effects caused by the setter thread prior to them
+//! calling `set` or `get_or_init` will be made visible to that thread; without
+//! it, it's possible for it to appear as if they haven't happened yet from the
+//! getter thread's perspective. This is an acceptable tradeoff to make since
+//! `Acquire` and `Release` have very little performance overhead on most
+//! architectures versus `Relaxed`.
+
+// The "atomic orderings" section of the documentation above promises
+// "happens-before" semantics. This drives the choice of orderings in the uses
+// of `compare_exchange` below. On success, the value was zero/null, so there
+// was nothing to acquire (there is never any `Ordering::Release` store of 0).
+// On failure, the value was nonzero, so it was initialized previously (perhaps
+// on another thread) using `Ordering::Release`, so we must use
+// `Ordering::Acquire` to ensure that store "happens-before" this load.
+
+#[cfg(not(feature = "portable-atomic"))]
+use core::sync::atomic;
+#[cfg(feature = "portable-atomic")]
+use portable_atomic as atomic;
+
+use atomic::{AtomicPtr, AtomicUsize, Ordering};
+use core::cell::UnsafeCell;
+use core::marker::PhantomData;
+use core::num::NonZeroUsize;
+use core::ptr;
+
+/// A thread-safe cell which can be written to only once.
+#[derive(Default, Debug)]
+pub struct OnceNonZeroUsize {
+ inner: AtomicUsize,
+}
+
+impl OnceNonZeroUsize {
+ /// Creates a new empty cell.
+ #[inline]
+ pub const fn new() -> Self {
+ Self { inner: AtomicUsize::new(0) }
+ }
+
+ /// Gets the underlying value.
+ #[inline]
+ pub fn get(&self) -> Option<NonZeroUsize> {
+ let val = self.inner.load(Ordering::Acquire);
+ NonZeroUsize::new(val)
+ }
+
+ /// Get the reference to the underlying value, without checking if the cell
+ /// is initialized.
+ ///
+ /// # Safety
+ ///
+ /// Caller must ensure that the cell is in initialized state, and that
+ /// the contents are acquired by (synchronized to) this thread.
+ pub unsafe fn get_unchecked(&self) -> NonZeroUsize {
+ #[inline(always)]
+ fn as_const_ptr(r: &AtomicUsize) -> *const usize {
+ use core::mem::align_of;
+
+ let p: *const AtomicUsize = r;
+ // SAFETY: "This type has the same size and bit validity as
+ // the underlying integer type, usize. However, the alignment of
+ // this type is always equal to its size, even on targets where
+ // usize has a lesser alignment."
+ const _ALIGNMENT_COMPATIBLE: () =
+ assert!(align_of::<AtomicUsize>() % align_of::<usize>() == 0);
+ p.cast::<usize>()
+ }
+
+ // TODO(MSRV-1.70): Use `AtomicUsize::as_ptr().cast_const()`
+ // See https://github.com/rust-lang/rust/issues/138246.
+ let p = as_const_ptr(&self.inner);
+
+ // SAFETY: The caller is responsible for ensuring that the value
+ // was initialized and that the contents have been acquired by
+ // this thread. Assuming that, we can assume there will be no
+ // conflicting writes to the value since the value will never
+ // change once initialized. This relies on the statement in
+ // https://doc.rust-lang.org/1.83.0/core/sync/atomic/ that "(A
+ // `compare_exchange` or `compare_exchange_weak` that does not
+ // succeed is not considered a write."
+ let val = unsafe { p.read() };
+
+ // SAFETY: The caller is responsible for ensuring the value is
+ // initialized and thus not zero.
+ unsafe { NonZeroUsize::new_unchecked(val) }
+ }
+
+ /// Sets the contents of this cell to `value`.
+ ///
+ /// Returns `Ok(())` if the cell was empty and `Err(())` if it was
+ /// full.
+ #[inline]
+ pub fn set(&self, value: NonZeroUsize) -> Result<(), ()> {
+ match self.compare_exchange(value) {
+ Ok(_) => Ok(()),
+ Err(_) => Err(()),
+ }
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if the cell was
+ /// empty.
+ ///
+ /// If several threads concurrently run `get_or_init`, more than one `f` can
+ /// be called. However, all threads will return the same value, produced by
+ /// some `f`.
+ pub fn get_or_init<F>(&self, f: F) -> NonZeroUsize
+ where
+ F: FnOnce() -> NonZeroUsize,
+ {
+ enum Void {}
+ match self.get_or_try_init(|| Ok::<NonZeroUsize, Void>(f())) {
+ Ok(val) => val,
+ Err(void) => match void {},
+ }
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if
+ /// the cell was empty. If the cell was empty and `f` failed, an
+ /// error is returned.
+ ///
+ /// If several threads concurrently run `get_or_init`, more than one `f` can
+ /// be called. However, all threads will return the same value, produced by
+ /// some `f`.
+ pub fn get_or_try_init<F, E>(&self, f: F) -> Result<NonZeroUsize, E>
+ where
+ F: FnOnce() -> Result<NonZeroUsize, E>,
+ {
+ match self.get() {
+ Some(it) => Ok(it),
+ None => self.init(f),
+ }
+ }
+
+ #[cold]
+ #[inline(never)]
+ fn init<E>(&self, f: impl FnOnce() -> Result<NonZeroUsize, E>) -> Result<NonZeroUsize, E> {
+ let nz = f()?;
+ let mut val = nz.get();
+ if let Err(old) = self.compare_exchange(nz) {
+ val = old;
+ }
+ Ok(unsafe { NonZeroUsize::new_unchecked(val) })
+ }
+
+ #[inline(always)]
+ fn compare_exchange(&self, val: NonZeroUsize) -> Result<usize, usize> {
+ self.inner.compare_exchange(0, val.get(), Ordering::Release, Ordering::Acquire)
+ }
+}
+
+/// A thread-safe cell which can be written to only once.
+#[derive(Default, Debug)]
+pub struct OnceBool {
+ inner: OnceNonZeroUsize,
+}
+
+impl OnceBool {
+ /// Creates a new empty cell.
+ #[inline]
+ pub const fn new() -> Self {
+ Self { inner: OnceNonZeroUsize::new() }
+ }
+
+ /// Gets the underlying value.
+ #[inline]
+ pub fn get(&self) -> Option<bool> {
+ self.inner.get().map(Self::from_usize)
+ }
+
+ /// Sets the contents of this cell to `value`.
+ ///
+ /// Returns `Ok(())` if the cell was empty and `Err(())` if it was
+ /// full.
+ #[inline]
+ pub fn set(&self, value: bool) -> Result<(), ()> {
+ self.inner.set(Self::to_usize(value))
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if the cell was
+ /// empty.
+ ///
+ /// If several threads concurrently run `get_or_init`, more than one `f` can
+ /// be called. However, all threads will return the same value, produced by
+ /// some `f`.
+ pub fn get_or_init<F>(&self, f: F) -> bool
+ where
+ F: FnOnce() -> bool,
+ {
+ Self::from_usize(self.inner.get_or_init(|| Self::to_usize(f())))
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if
+ /// the cell was empty. If the cell was empty and `f` failed, an
+ /// error is returned.
+ ///
+ /// If several threads concurrently run `get_or_init`, more than one `f` can
+ /// be called. However, all threads will return the same value, produced by
+ /// some `f`.
+ pub fn get_or_try_init<F, E>(&self, f: F) -> Result<bool, E>
+ where
+ F: FnOnce() -> Result<bool, E>,
+ {
+ self.inner.get_or_try_init(|| f().map(Self::to_usize)).map(Self::from_usize)
+ }
+
+ #[inline]
+ fn from_usize(value: NonZeroUsize) -> bool {
+ value.get() == 1
+ }
+
+ #[inline]
+ fn to_usize(value: bool) -> NonZeroUsize {
+ unsafe { NonZeroUsize::new_unchecked(if value { 1 } else { 2 }) }
+ }
+}
+
+/// A thread-safe cell which can be written to only once.
+pub struct OnceRef<'a, T> {
+ inner: AtomicPtr<T>,
+ ghost: PhantomData<UnsafeCell<&'a T>>,
+}
+
+// TODO: Replace UnsafeCell with SyncUnsafeCell once stabilized
+unsafe impl<'a, T: Sync> Sync for OnceRef<'a, T> {}
+
+impl<'a, T> core::fmt::Debug for OnceRef<'a, T> {
+ fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
+ write!(f, "OnceRef({:?})", self.inner)
+ }
+}
+
+impl<'a, T> Default for OnceRef<'a, T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<'a, T> OnceRef<'a, T> {
+ /// Creates a new empty cell.
+ pub const fn new() -> Self {
+ Self { inner: AtomicPtr::new(ptr::null_mut()), ghost: PhantomData }
+ }
+
+ /// Gets a reference to the underlying value.
+ pub fn get(&self) -> Option<&'a T> {
+ let ptr = self.inner.load(Ordering::Acquire);
+ unsafe { ptr.as_ref() }
+ }
+
+ /// Sets the contents of this cell to `value`.
+ ///
+ /// Returns `Ok(())` if the cell was empty and `Err(value)` if it was
+ /// full.
+ pub fn set(&self, value: &'a T) -> Result<(), ()> {
+ match self.compare_exchange(value) {
+ Ok(_) => Ok(()),
+ Err(_) => Err(()),
+ }
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if the cell was
+ /// empty.
+ ///
+ /// If several threads concurrently run `get_or_init`, more than one `f` can
+ /// be called. However, all threads will return the same value, produced by
+ /// some `f`.
+ pub fn get_or_init<F>(&self, f: F) -> &'a T
+ where
+ F: FnOnce() -> &'a T,
+ {
+ enum Void {}
+ match self.get_or_try_init(|| Ok::<&'a T, Void>(f())) {
+ Ok(val) => val,
+ Err(void) => match void {},
+ }
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if
+ /// the cell was empty. If the cell was empty and `f` failed, an
+ /// error is returned.
+ ///
+ /// If several threads concurrently run `get_or_init`, more than one `f` can
+ /// be called. However, all threads will return the same value, produced by
+ /// some `f`.
+ pub fn get_or_try_init<F, E>(&self, f: F) -> Result<&'a T, E>
+ where
+ F: FnOnce() -> Result<&'a T, E>,
+ {
+ match self.get() {
+ Some(val) => Ok(val),
+ None => self.init(f),
+ }
+ }
+
+ #[cold]
+ #[inline(never)]
+ fn init<E>(&self, f: impl FnOnce() -> Result<&'a T, E>) -> Result<&'a T, E> {
+ let mut value: &'a T = f()?;
+ if let Err(old) = self.compare_exchange(value) {
+ value = unsafe { &*old };
+ }
+ Ok(value)
+ }
+
+ #[inline(always)]
+ fn compare_exchange(&self, value: &'a T) -> Result<(), *const T> {
+ self.inner
+ .compare_exchange(
+ ptr::null_mut(),
+ <*const T>::cast_mut(value),
+ Ordering::Release,
+ Ordering::Acquire,
+ )
+ .map(|_: *mut T| ())
+ .map_err(<*mut T>::cast_const)
+ }
+
+ /// ```compile_fail
+ /// use once_cell::race::OnceRef;
+ ///
+ /// let mut l = OnceRef::new();
+ ///
+ /// {
+ /// let y = 2;
+ /// let mut r = OnceRef::new();
+ /// r.set(&y).unwrap();
+ /// core::mem::swap(&mut l, &mut r);
+ /// }
+ ///
+ /// // l now contains a dangling reference to y
+ /// eprintln!("uaf: {}", l.get().unwrap());
+ /// ```
+ fn _dummy() {}
+}
+
+#[cfg(feature = "alloc")]
+pub use self::once_box::OnceBox;
+
+#[cfg(feature = "alloc")]
+mod once_box {
+ use super::atomic::{AtomicPtr, Ordering};
+ use core::{marker::PhantomData, ptr};
+
+ use alloc::boxed::Box;
+
+ /// A thread-safe cell which can be written to only once.
+ pub struct OnceBox<T> {
+ inner: AtomicPtr<T>,
+ ghost: PhantomData<Option<Box<T>>>,
+ }
+
+ impl<T> core::fmt::Debug for OnceBox<T> {
+ fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
+ write!(f, "OnceBox({:?})", self.inner.load(Ordering::Relaxed))
+ }
+ }
+
+ impl<T> Default for OnceBox<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+ }
+
+ impl<T> Drop for OnceBox<T> {
+ fn drop(&mut self) {
+ let ptr = *self.inner.get_mut();
+ if !ptr.is_null() {
+ drop(unsafe { Box::from_raw(ptr) })
+ }
+ }
+ }
+
+ impl<T> OnceBox<T> {
+ /// Creates a new empty cell.
+ pub const fn new() -> Self {
+ Self { inner: AtomicPtr::new(ptr::null_mut()), ghost: PhantomData }
+ }
+
+ /// Creates a new cell with the given value.
+ pub fn with_value(value: Box<T>) -> Self {
+ Self { inner: AtomicPtr::new(Box::into_raw(value)), ghost: PhantomData }
+ }
+
+ /// Gets a reference to the underlying value.
+ pub fn get(&self) -> Option<&T> {
+ let ptr = self.inner.load(Ordering::Acquire);
+ if ptr.is_null() {
+ return None;
+ }
+ Some(unsafe { &*ptr })
+ }
+
+ /// Sets the contents of this cell to `value`.
+ ///
+ /// Returns `Ok(())` if the cell was empty and `Err(value)` if it was
+ /// full.
+ pub fn set(&self, value: Box<T>) -> Result<(), Box<T>> {
+ let ptr = Box::into_raw(value);
+ let exchange = self.inner.compare_exchange(
+ ptr::null_mut(),
+ ptr,
+ Ordering::Release,
+ Ordering::Acquire,
+ );
+ if exchange.is_err() {
+ let value = unsafe { Box::from_raw(ptr) };
+ return Err(value);
+ }
+ Ok(())
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if the cell was
+ /// empty.
+ ///
+ /// If several threads concurrently run `get_or_init`, more than one `f` can
+ /// be called. However, all threads will return the same value, produced by
+ /// some `f`.
+ pub fn get_or_init<F>(&self, f: F) -> &T
+ where
+ F: FnOnce() -> Box<T>,
+ {
+ enum Void {}
+ match self.get_or_try_init(|| Ok::<Box<T>, Void>(f())) {
+ Ok(val) => val,
+ Err(void) => match void {},
+ }
+ }
+
+ /// Gets the contents of the cell, initializing it with `f` if
+ /// the cell was empty. If the cell was empty and `f` failed, an
+ /// error is returned.
+ ///
+ /// If several threads concurrently run `get_or_init`, more than one `f` can
+ /// be called. However, all threads will return the same value, produced by
+ /// some `f`.
+ pub fn get_or_try_init<F, E>(&self, f: F) -> Result<&T, E>
+ where
+ F: FnOnce() -> Result<Box<T>, E>,
+ {
+ match self.get() {
+ Some(val) => Ok(val),
+ None => self.init(f)
+ }
+ }
+
+ #[cold]
+ #[inline(never)]
+ fn init<E>(&self, f: impl FnOnce() -> Result<Box<T>, E>) -> Result<&T, E> {
+ let val = f()?;
+ let mut ptr = Box::into_raw(val);
+ let exchange = self.inner.compare_exchange(
+ ptr::null_mut(),
+ ptr,
+ Ordering::Release,
+ Ordering::Acquire,
+ );
+ if let Err(old) = exchange {
+ drop(unsafe { Box::from_raw(ptr) });
+ ptr = old;
+ }
+ Ok(unsafe { &*ptr })
+ }
+ }
+
+ unsafe impl<T: Sync + Send> Sync for OnceBox<T> {}
+
+ impl<T: Clone> Clone for OnceBox<T> {
+ fn clone(&self) -> Self {
+ match self.get() {
+ Some(value) => OnceBox::with_value(Box::new(value.clone())),
+ None => OnceBox::new(),
+ }
+ }
+ }
+
+ /// ```compile_fail
+ /// struct S(*mut ());
+ /// unsafe impl Sync for S {}
+ ///
+ /// fn share<T: Sync>(_: &T) {}
+ /// share(&once_cell::race::OnceBox::<S>::new());
+ /// ```
+ fn _dummy() {}
+}
### external/vendor/once_cell/tests/it/main.rs
@@ -0,0 +1,12 @@
+mod unsync_once_cell;
+#[cfg(any(feature = "std", feature = "critical-section"))]
+mod sync_once_cell;
+
+mod unsync_lazy;
+#[cfg(any(feature = "std", feature = "critical-section"))]
+mod sync_lazy;
+
+#[cfg(feature = "race")]
+mod race;
+#[cfg(all(feature = "race", feature = "alloc"))]
+mod race_once_box;
### external/vendor/once_cell/tests/it/race.rs
@@ -0,0 +1,191 @@
+#[cfg(feature = "std")]
+use std::sync::Barrier;
+use std::{
+ num::NonZeroUsize,
+ sync::atomic::{AtomicUsize, Ordering::SeqCst},
+ thread::scope,
+};
+
+use once_cell::race::{OnceBool, OnceNonZeroUsize, OnceRef};
+
+#[test]
+fn once_non_zero_usize_smoke_test() {
+ let cnt = AtomicUsize::new(0);
+ let cell = OnceNonZeroUsize::new();
+ let val = NonZeroUsize::new(92).unwrap();
+ scope(|s| {
+ s.spawn(|| {
+ assert_eq!(
+ cell.get_or_init(|| {
+ cnt.fetch_add(1, SeqCst);
+ val
+ }),
+ val
+ );
+ assert_eq!(cnt.load(SeqCst), 1);
+
+ assert_eq!(
+ cell.get_or_init(|| {
+ cnt.fetch_add(1, SeqCst);
+ val
+ }),
+ val
+ );
+ assert_eq!(cnt.load(SeqCst), 1);
+ });
+ });
+ assert_eq!(cell.get(), Some(val));
+ assert_eq!(cnt.load(SeqCst), 1);
+}
+
+#[test]
+fn once_non_zero_usize_set() {
+ let val1 = NonZeroUsize::new(92).unwrap();
+ let val2 = NonZeroUsize::new(62).unwrap();
+
+ let cell = OnceNonZeroUsize::new();
+
+ assert!(cell.set(val1).is_ok());
+ assert_eq!(cell.get(), Some(val1));
+
+ assert!(cell.set(val2).is_err());
+ assert_eq!(cell.get(), Some(val1));
+}
+
+#[cfg(feature = "std")]
+#[test]
+fn once_non_zero_usize_first_wins() {
+ let val1 = NonZeroUsize::new(92).unwrap();
+ let val2 = NonZeroUsize::new(62).unwrap();
+
+ let cell = OnceNonZeroUsize::new();
+
+ let b1 = Barrier::new(2);
+ let b2 = Barrier::new(2);
+ let b3 = Barrier::new(2);
+ scope(|s| {
+ s.spawn(|| {
+ let r1 = cell.get_or_init(|| {
+ b1.wait();
+ b2.wait();
+ val1
+ });
+ assert_eq!(r1, val1);
+ b3.wait();
+ });
+ b1.wait();
+ s.spawn(|| {
+ let r2 = cell.get_or_init(|| {
+ b2.wait();
+ b3.wait();
+ val2
+ });
+ assert_eq!(r2, val1);
+ });
+ });
+
+ assert_eq!(cell.get(), Some(val1));
+}
+
+#[test]
+fn once_bool_smoke_test() {
+ let cnt = AtomicUsize::new(0);
+ let cell = OnceBool::new();
+ scope(|s| {
+ s.spawn(|| {
+ assert_eq!(
+ cell.get_or_init(|| {
+ cnt.fetch_add(1, SeqCst);
+ false
+ }),
+ false
+ );
+ assert_eq!(cnt.load(SeqCst), 1);
+
+ assert_eq!(
+ cell.get_or_init(|| {
+ cnt.fetch_add(1, SeqCst);
+ false
+ }),
+ false
+ );
+ assert_eq!(cnt.load(SeqCst), 1);
+ });
+ });
+ assert_eq!(cell.get(), Some(false));
+ assert_eq!(cnt.load(SeqCst), 1);
+}
+
+#[test]
+fn once_bool_set() {
+ let cell = OnceBool::new();
+
+ assert!(cell.set(false).is_ok());
+ assert_eq!(cell.get(), Some(false));
+
+ assert!(cell.set(true).is_err());
+ assert_eq!(cell.get(), Some(false));
+}
+
+#[test]
+fn once_bool_get_or_try_init() {
+ let cell = OnceBool::new();
+
+ let result1: Result<bool, ()> = cell.get_or_try_init(|| Ok(true));
+ let result2: Result<bool, ()> = cell.get_or_try_init(|| Ok(false));
+ assert_eq!(result1, Ok(true));
+ assert_eq!(result2, Ok(true));
+
+ let cell = OnceBool::new();
+
+ let result3: Result<bool, ()> = cell.get_or_try_init(|| Err(()));
+ assert_eq!(result3, Err(()));
+}
+
+#[test]
+fn once_ref_smoke_test() {
+ let cnt: AtomicUsize = AtomicUsize::new(0);
+ let cell: OnceRef<'_, &str> = OnceRef::new();
+ scope(|s| {
+ s.spawn(|| {
+ assert_eq!(
+ cell.get_or_init(|| {
+ cnt.fetch_add(1, SeqCst);
+ &"false"
+ }),
+ &"false"
+ );
+ assert_eq!(cnt.load(SeqCst), 1);
+
+ assert_eq!(
+ cell.get_or_init(|| {
+ cnt.fetch_add(1, SeqCst);
+ &"false"
+ }),
+ &"false"
+ );
+ assert_eq!(cnt.load(SeqCst), 1);
+ });
+ });
+ assert_eq!(cell.get(), Some(&"false"));
+ assert_eq!(cnt.load(SeqCst), 1);
+}
+
+#[test]
+fn once_ref_set() {
+ let cell: OnceRef<'_, &str> = OnceRef::new();
+
+ assert!(cell.set(&"false").is_ok());
+ assert_eq!(cell.get(), Some(&"false"));
+
+ assert!(cell.set(&"true").is_err());
+ assert_eq!(cell.get(), Some(&"false"));
+}
+
+#[test]
+fn get_unchecked() {
+ let cell = OnceNonZeroUsize::new();
+ cell.set(NonZeroUsize::new(92).unwrap()).unwrap();
+ let value = unsafe { cell.get_unchecked() };
+ assert_eq!(value, NonZeroUsize::new(92).unwrap());
+}
### external/vendor/once_cell/tests/it/race_once_box.rs
@@ -0,0 +1,162 @@
+#[cfg(feature = "std")]
+use std::sync::Barrier;
+use std::sync::{
+ atomic::{AtomicUsize, Ordering::SeqCst},
+ Arc,
+};
+
+use once_cell::race::OnceBox;
+
+#[derive(Default)]
+struct Heap {
+ total: Arc<AtomicUsize>,
+}
+
+#[derive(Debug)]
+struct Pebble<T> {
+ val: T,
+ total: Arc<AtomicUsize>,
+}
+
+impl<T> Drop for Pebble<T> {
+ fn drop(&mut self) {
+ self.total.fetch_sub(1, SeqCst);
+ }
+}
+
+impl Heap {
+ fn total(&self) -> usize {
+ self.total.load(SeqCst)
+ }
+ fn new_pebble<T>(&self, val: T) -> Pebble<T> {
+ self.total.fetch_add(1, SeqCst);
+ Pebble { val, total: Arc::clone(&self.total) }
+ }
+}
+
+#[cfg(feature = "std")]
+#[test]
+fn once_box_smoke_test() {
+ use std::thread::scope;
+
+ let heap = Heap::default();
+ let global_cnt = AtomicUsize::new(0);
+ let cell = OnceBox::new();
+ let b = Barrier::new(128);
+ scope(|s| {
+ for _ in 0..128 {
+ s.spawn(|| {
+ let local_cnt = AtomicUsize::new(0);
+ cell.get_or_init(|| {
+ global_cnt.fetch_add(1, SeqCst);
+ local_cnt.fetch_add(1, SeqCst);
+ b.wait();
+ Box::new(heap.new_pebble(()))
+ });
+ assert_eq!(local_cnt.load(SeqCst), 1);
+
+ cell.get_or_init(|| {
+ global_cnt.fetch_add(1, SeqCst);
+ local_cnt.fetch_add(1, SeqCst);
+ Box::new(heap.new_pebble(()))
+ });
+ assert_eq!(local_cnt.load(SeqCst), 1);
+ });
+ }
+ });
+ assert!(cell.get().is_some());
+ assert!(global_cnt.load(SeqCst) > 10);
+
+ assert_eq!(heap.total(), 1);
+ drop(cell);
+ assert_eq!(heap.total(), 0);
+}
+
+#[test]
+fn once_box_set() {
+ let heap = Heap::default();
+ let cell = OnceBox::new();
+ assert!(cell.get().is_none());
+
+ assert!(cell.set(Box::new(heap.new_pebble("hello"))).is_ok());
+ assert_eq!(cell.get().unwrap().val, "hello");
+ assert_eq!(heap.total(), 1);
+
+ assert!(cell.set(Box::new(heap.new_pebble("world"))).is_err());
+ assert_eq!(cell.get().unwrap().val, "hello");
+ assert_eq!(heap.total(), 1);
+
+ drop(cell);
+ assert_eq!(heap.total(), 0);
+}
+
+#[cfg(feature = "std")]
+#[test]
+fn once_box_first_wins() {
+ use std::thread::scope;
+
+ let cell = OnceBox::new();
+ let val1 = 92;
+ let val2 = 62;
+
+ let b1 = Barrier::new(2);
+ let b2 = Barrier::new(2);
+ let b3 = Barrier::new(2);
+ scope(|s| {
+ s.spawn(|| {
+ let r1 = cell.get_or_init(|| {
+ b1.wait();
+ b2.wait();
+ Box::new(val1)
+ });
+ assert_eq!(*r1, val1);
+ b3.wait();
+ });
+ b1.wait();
+ s.spawn(|| {
+ let r2 = cell.get_or_init(|| {
+ b2.wait();
+ b3.wait();
+ Box::new(val2)
+ });
+ assert_eq!(*r2, val1);
+ });
+ });
+
+ assert_eq!(cell.get(), Some(&val1));
+}
+
+#[test]
+fn once_box_reentrant() {
+ let cell = OnceBox::new();
+ let res = cell.get_or_init(|| {
+ cell.get_or_init(|| Box::new("hello".to_string()));
+ Box::new("world".to_string())
+ });
+ assert_eq!(res, "hello");
+}
+
+#[test]
+fn once_box_default() {
+ struct Foo;
+
+ let cell: OnceBox<Foo> = Default::default();
+ assert!(cell.get().is_none());
+}
+
+#[test]
+fn onece_box_with_value() {
+ let cell = OnceBox::with_value(Box::new(92));
+ assert_eq!(cell.get(), Some(&92));
+}
+
+#[test]
+fn onece_box_clone() {
+ let cell1 = OnceBox::new();
+ let cell2 = cell1.clone();
+ cell1.set(Box::new(92)).unwrap();
+ let cell3 = cell1.clone();
+ assert_eq!(cell1.get(), Some(&92));
+ assert_eq!(cell2.get(), None);
+ assert_eq!(cell3.get(), Some(&92));
+}
### external/vendor/once_cell/tests/it/sync_lazy.rs
@@ -0,0 +1,176 @@
+use std::{
+ cell::Cell,
+ sync::atomic::{AtomicUsize, Ordering::SeqCst},
+ thread::scope,
+};
+
+use once_cell::sync::{Lazy, OnceCell};
+
+#[test]
+fn lazy_new() {
+ let called = AtomicUsize::new(0);
+ let x = Lazy::new(|| {
+ called.fetch_add(1, SeqCst);
+ 92
+ });
+
+ assert_eq!(called.load(SeqCst), 0);
+
+ scope(|s| {
+ s.spawn(|| {
+ let y = *x - 30;
+ assert_eq!(y, 62);
+ assert_eq!(called.load(SeqCst), 1);
+ });
+ });
+
+ let y = *x - 30;
+ assert_eq!(y, 62);
+ assert_eq!(called.load(SeqCst), 1);
+}
+
+#[test]
+fn lazy_deref_mut() {
+ let called = AtomicUsize::new(0);
+ let mut x = Lazy::new(|| {
+ called.fetch_add(1, SeqCst);
+ 92
+ });
+
+ assert_eq!(called.load(SeqCst), 0);
+
+ let y = *x - 30;
+ assert_eq!(y, 62);
+ assert_eq!(called.load(SeqCst), 1);
+
+ *x /= 2;
+ assert_eq!(*x, 46);
+ assert_eq!(called.load(SeqCst), 1);
+}
+
+#[test]
+fn lazy_force_mut() {
+ let called = Cell::new(0);
+ let mut x = Lazy::new(|| {
+ called.set(called.get() + 1);
+ 92
+ });
+ assert_eq!(called.get(), 0);
+ let v = Lazy::force_mut(&mut x);
+ assert_eq!(called.get(), 1);
+
+ *v /= 2;
+ assert_eq!(*x, 46);
+ assert_eq!(called.get(), 1);
+}
+
+#[test]
+fn lazy_get_mut() {
+ let called = Cell::new(0);
+ let mut x: Lazy<u32, _> = Lazy::new(|| {
+ called.set(called.get() + 1);
+ 92
+ });
+
+ assert_eq!(called.get(), 0);
+ assert_eq!(*x, 92);
+
+ let mut_ref: &mut u32 = Lazy::get_mut(&mut x).unwrap();
+ assert_eq!(called.get(), 1);
+
+ *mut_ref /= 2;
+ assert_eq!(*x, 46);
+ assert_eq!(called.get(), 1);
+}
+
+#[test]
+fn lazy_default() {
+ static CALLED: AtomicUsize = AtomicUsize::new(0);
+
+ struct Foo(u8);
+ impl Default for Foo {
+ fn default() -> Self {
+ CALLED.fetch_add(1, SeqCst);
+ Foo(42)
+ }
+ }
+
+ let lazy: Lazy<std::sync::Mutex<Foo>> = <_>::default();
+
+ assert_eq!(CALLED.load(SeqCst), 0);
+
+ assert_eq!(lazy.lock().unwrap().0, 42);
+ assert_eq!(CALLED.load(SeqCst), 1);
+
+ lazy.lock().unwrap().0 = 21;
+
+ assert_eq!(lazy.lock().unwrap().0, 21);
+ assert_eq!(CALLED.load(SeqCst), 1);
+}
+
+#[test]
+fn static_lazy() {
+ static XS: Lazy<Vec<i32>> = Lazy::new(|| {
+ let mut xs = Vec::new();
+ xs.push(1);
+ xs.push(2);
+ xs.push(3);
+ xs
+ });
+ scope(|s| {
+ s.spawn(|| {
+ assert_eq!(&*XS, &vec![1, 2, 3]);
+ });
+ });
+ assert_eq!(&*XS, &vec![1, 2, 3]);
+}
+
+#[test]
+fn static_lazy_via_fn() {
+ fn xs() -> &'static Vec<i32> {
+ static XS: OnceCell<Vec<i32>> = OnceCell::new();
+ XS.get_or_init(|| {
+ let mut xs = Vec::new();
+ xs.push(1);
+ xs.push(2);
+ xs.push(3);
+ xs
+ })
+ }
+ assert_eq!(xs(), &vec![1, 2, 3]);
+}
+
+#[test]
+fn lazy_into_value() {
+ let l: Lazy<i32, _> = Lazy::new(|| panic!());
+ assert!(matches!(Lazy::into_value(l), Err(_)));
+ let l = Lazy::new(|| -> i32 { 92 });
+ Lazy::force(&l);
+ assert!(matches!(Lazy::into_value(l), Ok(92)));
+}
+
+#[test]
+fn lazy_poisoning() {
+ let x: Lazy<String> = Lazy::new(|| panic!("kaboom"));
+ for _ in 0..2 {
+ let res = std::panic::catch_unwind(|| x.len());
+ assert!(res.is_err());
+ }
+}
+
+#[test]
+// https://github.com/rust-lang/rust/issues/34761#issuecomment-256320669
+fn arrrrrrrrrrrrrrrrrrrrrr() {
+ let lazy: Lazy<&String, _>;
+ {
+ let s = String::new();
+ lazy = Lazy::new(|| &s);
+ _ = *lazy;
+ }
+}
+
+#[test]
+fn lazy_is_sync_send() {
+ fn assert_traits<T: Send + Sync>() {}
+ assert_traits::<Lazy<String>>();
+}
### external/vendor/once_cell/tests/it/sync_once_cell.rs
@@ -0,0 +1,328 @@
+use std::{
+ sync::atomic::{AtomicUsize, Ordering::SeqCst},
+ thread::scope,
+};
+
+#[cfg(feature = "std")]
+use std::sync::Barrier;
+
+#[cfg(not(feature = "std"))]
+use core::cell::Cell;
+
+use once_cell::sync::{Lazy, OnceCell};
+
+#[test]
+fn once_cell() {
+ let c = OnceCell::new();
+ assert!(c.get().is_none());
+ scope(|s| {
+ s.spawn(|| {
+ c.get_or_init(|| 92);
+ assert_eq!(c.get(), Some(&92));
+ });
+ });
+ c.get_or_init(|| panic!("Kabom!"));
+ assert_eq!(c.get(), Some(&92));
+}
+
+#[test]
+fn once_cell_with_value() {
+ static CELL: OnceCell<i32> = OnceCell::with_value(12);
+ assert_eq!(CELL.get(), Some(&12));
+}
+
+#[test]
+fn once_cell_get_mut() {
+ let mut c = OnceCell::new();
+ assert!(c.get_mut().is_none());
+ c.set(90).unwrap();
+ *c.get_mut().unwrap() += 2;
+ assert_eq!(c.get_mut(), Some(&mut 92));
+}
+
+#[test]
+fn once_cell_get_unchecked() {
+ let c = OnceCell::new();
+ c.set(92).unwrap();
+ unsafe {
+ assert_eq!(c.get_unchecked(), &92);
+ }
+}
+
+#[test]
+fn once_cell_drop() {
+ static DROP_CNT: AtomicUsize = AtomicUsize::new(0);
+ struct Dropper;
+ impl Drop for Dropper {
+ fn drop(&mut self) {
+ DROP_CNT.fetch_add(1, SeqCst);
+ }
+ }
+
+ let x = OnceCell::new();
+ scope(|s| {
+ s.spawn(|| {
+ x.get_or_init(|| Dropper);
+ assert_eq!(DROP_CNT.load(SeqCst), 0);
+ drop(x);
+ });
+ });
+ assert_eq!(DROP_CNT.load(SeqCst), 1);
+}
+
+#[test]
+fn once_cell_drop_empty() {
+ let x = OnceCell::<String>::new();
+ drop(x);
+}
+
+#[test]
+fn clone() {
+ let s = OnceCell::new();
+ let c = s.clone();
+ assert!(c.get().is_none());
+
+ s.set("hello".to_string()).unwrap();
+ let c = s.clone();
+ assert_eq!(c.get().map(String::as_str), Some("hello"));
+}
+
+#[test]
+fn get_or_try_init() {
+ let cell: OnceCell<String> = OnceCell::new();
+ assert!(cell.get().is_none());
+
+ let res = std::panic::catch_unwind(|| cell.get_or_try_init(|| -> Result<_, ()> { panic!() }));
+ assert!(res.is_err());
+ assert!(cell.get().is_none());
+
+ assert_eq!(cell.get_or_try_init(|| Err(())), Err(()));
+
+ assert_eq!(cell.get_or_try_init(|| Ok::<_, ()>("hello".to_string())), Ok(&"hello".to_string()));
+ assert_eq!(cell.get(), Some(&"hello".to_string()));
+}
+
+#[cfg(feature = "std")]
+#[test]
+fn wait() {
+ let cell: OnceCell<String> = OnceCell::new();
+ scope(|s| {
+ s.spawn(|| cell.set("hello".to_string()));
+ let greeting = cell.wait();
+ assert_eq!(greeting, "hello")
+ });
+}
+
+#[cfg(feature = "std")]
+#[test]
+fn wait_panic() {
+ let cell: OnceCell<String> = OnceCell::new();
+ scope(|s| {
+ let h1 = s.spawn(|| {
+ cell.get_or_try_init(|| -> Result<String, ()> { panic!() }).unwrap();
+ });
+ let h2 = s.spawn(|| {
+ assert!(h1.join().is_err());
+ cell.get_or_try_init(|| -> Result<String, ()> { Ok("hello".to_string()) }).unwrap();
+ });
+
+ let greeting = cell.wait();
+ assert_eq!(greeting, "hello");
+ assert!(h2.join().is_ok());
+ });
+}
+
+#[cfg(feature = "std")]
+#[test]
+fn get_or_init_stress() {
+ let n_threads = if cfg!(miri) { 30 } else { 1_000 };
+ let n_cells = if cfg!(miri) { 30 } else { 1_000 };
+ let cells: Vec<_> = std::iter::repeat_with(|| (Barrier::new(n_threads), OnceCell::new()))
+ .take(n_cells)
+ .collect();
+ scope(|s| {
+ for t in 0..n_threads {
+ let cells = &cells;
+ s.spawn(move || {
+ for (i, (b, s)) in cells.iter().enumerate() {
+ b.wait();
+ let j = if t % 2 == 0 { s.wait() } else { s.get_or_init(|| i) };
+ assert_eq!(*j, i);
+ }
+ });
+ }
+ });
+}
+
+#[test]
+fn from_impl() {
+ assert_eq!(OnceCell::from("value").get(), Some(&"value"));
+ assert_ne!(OnceCell::from("foo").get(), Some(&"bar"));
+}
+
+#[test]
+fn partialeq_impl() {
+ assert!(OnceCell::from("value") == OnceCell::from("value"));
+ assert!(OnceCell::from("foo") != OnceCell::from("bar"));
+
+ assert!(OnceCell::<String>::new() == OnceCell::new());
+ assert!(OnceCell::<String>::new() != OnceCell::from("value".to_owned()));
+}
+
+#[test]
+fn into_inner() {
+ let cell: OnceCell<String> = OnceCell::new();
+ assert_eq!(cell.into_inner(), None);
+ let cell = OnceCell::new();
+ cell.set("hello".to_string()).unwrap();
+ assert_eq!(cell.into_inner(), Some("hello".to_string()));
+}
+
+#[test]
+fn debug_impl() {
+ let cell = OnceCell::new();
+ assert_eq!(format!("{:#?}", cell), "OnceCell(Uninit)");
+ cell.set(vec!["hello", "world"]).unwrap();
+ assert_eq!(
+ format!("{:#?}", cell),
+ r#"OnceCell(
+ [
+ "hello",
+ "world",
+ ],
+)"#
+ );
+}
+
+#[test]
+#[cfg_attr(miri, ignore)] // miri doesn't support processes
+#[cfg(feature = "std")]
+fn reentrant_init() {
+ let examples_dir = {
+ let mut exe = std::env::current_exe().unwrap();
+ exe.pop();
+ exe.pop();
+ exe.push("examples");
+ exe
+ };
+ let bin = examples_dir
+ .join("reentrant_init_deadlocks")
+ .with_extension(std::env::consts::EXE_EXTENSION);
+ let mut guard = Guard { child: std::process::Command::new(bin).spawn().unwrap() };
+ std::thread::sleep(std::time::Duration::from_secs(2));
+ let status = guard.child.try_wait().unwrap();
+ assert!(status.is_none());
+
+ struct Guard {
+ child: std::process::Child,
+ }
+
+ impl Drop for Guard {
+ fn drop(&mut self) {
+ let _ = self.child.kill();
+ }
+ }
+}
+
+#[cfg(not(feature = "std"))]
+#[test]
+#[should_panic(expected = "reentrant init")]
+fn reentrant_init() {
+ let x: OnceCell<Box<i32>> = OnceCell::new();
+ let dangling_ref: Cell<Option<&i32>> = Cell::new(None);
+ x.get_or_init(|| {
+ let r = x.get_or_init(|| Box::new(92));
+ dangling_ref.set(Some(r));
+ Box::new(62)
+ });
+ eprintln!("use after free: {:?}", dangling_ref.get().unwrap());
+}
+
+#[test]
+fn eval_once_macro() {
+ macro_rules! eval_once {
+ (|| -> $ty:ty {
+ $($body:tt)*
+ }) => {{
+ static ONCE_CELL: OnceCell<$ty> = OnceCell::new();
+ fn init() -> $ty {
+ $($body)*
+ }
+ ONCE_CELL.get_or_init(init)
+ }};
+ }
+
+ let fib: &'static Vec<i32> = eval_once! {
+ || -> Vec<i32> {
+ let mut res = vec![1, 1];
+ for i in 0..10 {
+ let next = res[i] + res[i + 1];
+ res.push(next);
+ }
+ res
+ }
+ };
+ assert_eq!(fib[5], 8)
+}
+
+#[test]
+fn once_cell_does_not_leak_partially_constructed_boxes() {
+ let n_tries = if cfg!(miri) { 10 } else { 100 };
+ let n_readers = 10;
+ let n_writers = 3;
+ const MSG: &str = "Hello, World";
+
+ for _ in 0..n_tries {
+ let cell: OnceCell<String> = OnceCell::new();
+ scope(|scope| {
+ for _ in 0..n_readers {
+ scope.spawn(|| loop {
+ if let Some(msg) = cell.get() {
+ assert_eq!(msg, MSG);
+ break;
+ }
+ });
+ }
+ for _ in 0..n_writers {
+ let _ = scope.spawn(|| cell.set(MSG.to_owned()));
+ }
+ });
+ }
+}
+
+#[cfg(feature = "std")]
+#[test]
+fn get_does_not_block() {
+ let cell = OnceCell::new();
+ let barrier = Barrier::new(2);
+ scope(|scope| {
+ scope.spawn(|| {
+ cell.get_or_init(|| {
+ barrier.wait();
+ barrier.wait();
+ "hello".to_string()
+ });
+ });
+ barrier.wait();
+ assert_eq!(cell.get(), None);
+ barrier.wait();
+ });
+ assert_eq!(cell.get(), Some(&"hello".to_string()));
+}
+
+#[test]
+// https://github.com/rust-lang/rust/issues/34761#issuecomment-256320669
+fn arrrrrrrrrrrrrrrrrrrrrr() {
+ let cell = OnceCell::new();
+ {
+ let s = String::new();
+ cell.set(&s).unwrap();
+ }
+}
+
+#[test]
+fn once_cell_is_sync_send() {
+ fn assert_traits<T: Send + Sync>() {}
+ assert_traits::<OnceCell<String>>();
+ assert_traits::<Lazy<String>>();
+}
### external/vendor/once_cell/tests/it/unsync_lazy.rs
@@ -0,0 +1,134 @@
+use core::{
+ cell::Cell,
+ sync::atomic::{AtomicUsize, Ordering::SeqCst},
+};
+
+use once_cell::unsync::Lazy;
+
+#[test]
+fn lazy_new() {
+ let called = Cell::new(0);
+ let x = Lazy::new(|| {
+ called.set(called.get() + 1);
+ 92
+ });
+
+ assert_eq!(called.get(), 0);
+
+ let y = *x - 30;
+ assert_eq!(y, 62);
+ assert_eq!(called.get(), 1);
+
+ let y = *x - 30;
+ assert_eq!(y, 62);
+ assert_eq!(called.get(), 1);
+}
+
+#[test]
+fn lazy_deref_mut() {
+ let called = Cell::new(0);
+ let mut x = Lazy::new(|| {
+ called.set(called.get() + 1);
+ 92
+ });
+
+ assert_eq!(called.get(), 0);
+
+ let y = *x - 30;
+ assert_eq!(y, 62);
+ assert_eq!(called.get(), 1);
+
+ *x /= 2;
+ assert_eq!(*x, 46);
+ assert_eq!(called.get(), 1);
+}
+
+#[test]
+fn lazy_force_mut() {
+ let called = Cell::new(0);
+ let mut x = Lazy::new(|| {
+ called.set(called.get() + 1);
+ 92
+ });
+ assert_eq!(called.get(), 0);
+ let v = Lazy::force_mut(&mut x);
+ assert_eq!(called.get(), 1);
+
+ *v /= 2;
+ assert_eq!(*x, 46);
+ assert_eq!(called.get(), 1);
+}
+
+#[test]
+fn lazy_get_mut() {
+ let called = Cell::new(0);
+ let mut x: Lazy<u32, _> = Lazy::new(|| {
+ called.set(called.get() + 1);
+ 92
+ });
+
+ assert_eq!(called.get(), 0);
+ assert_eq!(*x, 92);
+
+ let mut_ref: &mut u32 = Lazy::get_mut(&mut x).unwrap();
+ assert_eq!(called.get(), 1);
+
+ *mut_ref /= 2;
+ assert_eq!(*x, 46);
+ assert_eq!(called.get(), 1);
+}
+
+#[test]
+fn lazy_default() {
+ static CALLED: AtomicUsize = AtomicUsize::new(0);
+
+ struct Foo(u8);
+ impl Default for Foo {
+ fn default() -> Self {
+ CALLED.fetch_add(1, SeqCst);
+ Foo(42)
+ }
+ }
+
+ let lazy: Lazy<std::sync::Mutex<Foo>> = <_>::default();
+
+ assert_eq!(CALLED.load(SeqCst), 0);
+
+ assert_eq!(lazy.lock().unwrap().0, 42);
+ assert_eq!(CALLED.load(SeqCst), 1);
+
+ lazy.lock().unwrap().0 = 21;
+
+ assert_eq!(lazy.lock().unwrap().0, 21);
+ assert_eq!(CALLED.load(SeqCst), 1);
+}
+
+#[test]
+fn lazy_into_value() {
+ let l: Lazy<i32, _> = Lazy::new(|| panic!());
+ assert!(matches!(Lazy::into_value(l), Err(_)));
+ let l = Lazy::new(|| -> i32 { 92 });
+ Lazy::force(&l);
+ assert!(matches!(Lazy::into_value(l), Ok(92)));
+}
+
+#[test]
+#[cfg(feature = "std")]
+fn lazy_poisoning() {
+ let x: Lazy<String> = Lazy::new(|| panic!("kaboom"));
+ for _ in 0..2 {
+ let res = std::panic::catch_unwind(|| x.len());
+ assert!(res.is_err());
+ }
+}
+
+#[test]
+// https://github.com/rust-lang/rust/issues/34761#issuecomment-256320669
+fn arrrrrrrrrrrrrrrrrrrrrr() {
+ let lazy: Lazy<&String, _>;
+ {
+ let s = String::new();
+ lazy = Lazy::new(|| &s);
+ _ = *lazy;
+ }
+}
### external/vendor/once_cell/tests/it/unsync_once_cell.rs
@@ -0,0 +1,154 @@
+use core::{
+ cell::Cell,
+ sync::atomic::{AtomicUsize, Ordering::SeqCst},
+};
+
+use once_cell::unsync::OnceCell;
+
+#[test]
+fn once_cell() {
+ let c = OnceCell::new();
+ assert!(c.get().is_none());
+ c.get_or_init(|| 92);
+ assert_eq!(c.get(), Some(&92));
+
+ c.get_or_init(|| panic!("Kabom!"));
+ assert_eq!(c.get(), Some(&92));
+}
+
+#[test]
+fn once_cell_with_value() {
+ const CELL: OnceCell<i32> = OnceCell::with_value(12);
+ let cell = CELL;
+ assert_eq!(cell.get(), Some(&12));
+}
+
+#[test]
+fn once_cell_get_mut() {
+ let mut c = OnceCell::new();
+ assert!(c.get_mut().is_none());
+ c.set(90).unwrap();
+ *c.get_mut().unwrap() += 2;
+ assert_eq!(c.get_mut(), Some(&mut 92));
+}
+
+#[test]
+fn once_cell_drop() {
+ static DROP_CNT: AtomicUsize = AtomicUsize::new(0);
+ struct Dropper;
+ impl Drop for Dropper {
+ fn drop(&mut self) {
+ DROP_CNT.fetch_add(1, SeqCst);
+ }
+ }
+
+ let x = OnceCell::new();
+ x.get_or_init(|| Dropper);
+ assert_eq!(DROP_CNT.load(SeqCst), 0);
+ drop(x);
+ assert_eq!(DROP_CNT.load(SeqCst), 1);
+}
+
+#[test]
+fn once_cell_drop_empty() {
+ let x = OnceCell::<String>::new();
+ drop(x);
+}
+
+#[test]
+fn clone() {
+ let s = OnceCell::new();
+ let c = s.clone();
+ assert!(c.get().is_none());
+
+ s.set("hello".to_string()).unwrap();
+ let c = s.clone();
+ assert_eq!(c.get().map(String::as_str), Some("hello"));
+}
+
+#[test]
+fn get_or_try_init() {
+ let cell: OnceCell<String> = OnceCell::new();
+ assert!(cell.get().is_none());
+
+ let res = std::panic::catch_unwind(|| cell.get_or_try_init(|| -> Result<_, ()> { panic!() }));
+ assert!(res.is_err());
+ assert!(cell.get().is_none());
+
+ assert_eq!(cell.get_or_try_init(|| Err(())), Err(()));
+
+ assert_eq!(cell.get_or_try_init(|| Ok::<_, ()>("hello".to_string())), Ok(&"hello".to_string()));
+ assert_eq!(cell.get(), Some(&"hello".to_string()));
+}
+
+#[test]
+fn from_impl() {
+ assert_eq!(OnceCell::from("value").get(), Some(&"value"));
+ assert_ne!(OnceCell::from("foo").get(), Some(&"bar"));
+}
+
+#[test]
+fn partialeq_impl() {
+ assert!(OnceCell::from("value") == OnceCell::from("value"));
+ assert!(OnceCell::from("foo") != OnceCell::from("bar"));
+
+ assert!(OnceCell::<String>::new() == OnceCell::new());
+ assert!(OnceCell::<String>::new() != OnceCell::from("value".to_owned()));
+}
+
+#[test]
+fn into_inner() {
+ let cell: OnceCell<String> = OnceCell::new();
+ assert_eq!(cell.into_inner(), None);
+ let cell = OnceCell::new();
+ cell.set("hello".to_string()).unwrap();
+ assert_eq!(cell.into_inner(), Some("hello".to_string()));
+}
+
+#[test]
+fn debug_impl() {
+ let cell = OnceCell::new();
+ assert_eq!(format!("{:#?}", cell), "OnceCell(Uninit)");
+ cell.set(vec!["hello", "world"]).unwrap();
+ assert_eq!(
+ format!("{:#?}", cell),
+ r#"OnceCell(
+ [
+ "hello",
+ "world",
+ ],
+)"#
+ );
+}
+
+#[test]
+#[should_panic(expected = "reentrant init")]
+fn reentrant_init() {
+ let x: OnceCell<Box<i32>> = OnceCell::new();
+ let dangling_ref: Cell<Option<&i32>> = Cell::new(None);
+ x.get_or_init(|| {
+ let r = x.get_or_init(|| Box::new(92));
+ dangling_ref.set(Some(r));
+ Box::new(62)
+ });
+ eprintln!("use after free: {:?}", dangling_ref.get().unwrap());
+}
+
+#[test]
+fn aliasing_in_get() {
+ let x = OnceCell::new();
+ x.set(42).unwrap();
+ let at_x = x.get().unwrap(); // --- (shared) borrow of inner `Option<T>` --+
+ let _ = x.set(27); // <-- temporary (unique) borrow of inner `Option<T>` |
+ println!("{}", at_x); // <------- up until here ---------------------------+
+}
+
+#[test]
+// https://github.com/rust-lang/rust/issues/34761#issuecomment-256320669
+fn arrrrrrrrrrrrrrrrrrrrrr() {
+ let cell = OnceCell::new();
+ {
+ let s = String::new();
+ cell.set(&s).unwrap();
+ }
+}
### external/vendor/portable-atomic/.cargo-checksum.json
@@ -1 +1 @@
-{"files":{".cargo_vcs_info.json":"101e4d497d04df25c65bba8108aabcda4335798a54bd12e0b047d882632973af","CHANGELOG.md":"73510da9703f32810945f437bc6af7373c5e7b7baca836e0e9f55367400f5192","Cargo.toml":"59f964afd1544e564614bc835456783c49dda7fc597dc429b79cf32d6b1714d0","Cargo.toml.orig":"5d1d1e05e84e41d80d061f6bea3949505f4f117e9150f6ee21858c9bc3c882ab","LICENSE-APACHE":"0d542e0c8804e39aa7f37eb00da5a762149dc682d7829451287e11b938e94594","LICENSE-MIT":"23f18e03dc49df91622fe2a76176497404e46ced8a715d9d2b67a7446571cca3","README.md":"752da4699331c29a602cebc1b11cd142cf3a4d147cc9ef888218f5bfd88a4f5d","build.rs":"eb6f945cf6bc83d0785c00c102e5607fad76366f374ed392ed8360f93c287946","no_atomic.rs":"f3cbde2210b7275786334ac8baa6900da591f7dc72c1c15f958b059a4faec9e2","src/cfgs.rs":"3316d0be451271a309a5812787a3f7a8445a34262379019c648e552a578d60fb","src/gen/utils.rs":"c529eb0b5050ada586207fdb2b8efc93e3157fae4b9fbec0125b5e107b36f0ba","src/imp/atomic128/README.md":"d4a2ac079298a0f26e1b8ddcb31ee62a56d42925ba1574b8dcb1e48a017f28fc","src/imp/atomic128/aarch64.rs":"26c035b651c75d410e76b9eb02fefe98e2b2ad90bc99a7fc8b51243db368497f","src/imp/atomic128/intrinsics.rs":"92abf18893a1d9f418707b053f101dc3bfbd4f5855bef834ab0e743f7a7d2664","src/imp/atomic128/macros.rs":"31f63dd5f6523fdc80d752afdd7b13e90394e7210b8f14bfa38a4feab0822cd5","src/imp/atomic128/mod.rs":"e8f2e3a1b90019dd780451aa77dc2a7560e6b1c6ac9e2814809ec9dfe43f0227","src/imp/atomic128/powerpc64.rs":"e058ed44fd1520d60b2df5475b37fdb190a4b7478e72d377621c9e21d09b3f88","src/imp/atomic128/riscv64.rs":"e33c85500bf12e5acb7eb99924ede20781acc6da782549a611fc6f2611eb5f93","src/imp/atomic128/s390x.rs":"ed2393e15e4263944ac530966d30f04a3c2e0514c0af4cb9a8560d5631e2afe6","src/imp/atomic128/x86_64.rs":"9e83cf5352ec278e4fdf3373a324fce74055e8b0b40ee6e35d56bc9136ac04e9","src/imp/atomic64/README.md":"4eeaa7a0a8e2ca661128168b7e952a3dfe740713633cb7d164bbcc7f580343f0","src/imp/atomic64/arm_linux.rs":"5db69d12916105f1718fb3a10c0a2bf353f6f25f49a409732c12e7ba0d33bdac","src/imp/atomic64/macros.rs":"5b6f0bfe40064885926cfd57a336f15da92f6964864c50d339d1bb9903802a9d","src/imp/atomic64/mod.rs":"73aaefa7f802d2423a77fcd80c1de75ae051c49f30365b075853bf8f70df27a0","src/imp/atomic64/riscv32.rs":"b3398f20ffe75630ae46806e12f4c4f421fdb23c0836b0cea949d98a65ffa8fc","src/imp/avr.rs":"a8d2c906c363144c353521b7d16cc01fd5509759b5d15172e7987d24f1dce6ac","src/imp/core_atomic.rs":"6bee6ff9d9bb75ba127da0a75b7999b0529155cefe1811fbdab8888c7065fe5a","src/imp/detect/README.md":"8e876f111004092afa38f98290d7cbec319a0ac9122ff1391c3fbff75510100b","src/imp/detect/aarch64_aa64reg.rs":"a70a9f2cc791ebfbb76b3afd0496f958476be4ac2dd7b367381c3374c1897d82","src/imp/detect/aarch64_apple.rs":"71c2e8a551ac4005bbbe3143351686434cd0d19a8c4ad13233e0db608fb49d45","src/imp/detect/aarch64_fuchsia.rs":"a029544ca15e017dbe14fe359e112f2e1780f3f1b778f68b41cc457bc67eccda","src/imp/detect/aarch64_illumos.rs":"a3539100147e91e1e99857e939f84c4f7481e9dc7e31a9c994a1e1353e12e5df","src/imp/detect/aarch64_windows.rs":"eadfa926641d9f4d0aa2356a7609a18a4f82cdfeb339ff153c8bb9c33b6c4f16","src/imp/detect/auxv.rs":"90196f1c9f0dcd1f638248b296f9e1b55df2b18a93f579954af971bb65176eaa","src/imp/detect/common.rs":"f9539a0dab4b0c331909a85d960661f5c671f1069db0d23de1fc6f2165337738","src/imp/detect/riscv_linux.rs":"2dae5a66b88b413409cc836d0c1055597a2861235f2228e88f0fd0f8865a5339","src/imp/detect/x86_64.rs":"c9fd6c2b3c39432b1d6815e454bec06b26fb1f5d0d6127e5dc5d5641c0599107","src/imp/fallback/mod.rs":"8c5a34a2ba01ed4af20d857d594c50d8e1a1bf88a7f506fe5cfd1c728b050e7c","src/imp/fallback/outline_atomics.rs":"56ff7b7d62c738be021d02d9d023829297a3b2e8b57964657d6faa7b135f0309","src/imp/fallback/seq_lock.rs":"b591a77f202095e31c855f61932145e85206a489ee1a0b23b89e4411ef02807e","src/imp/fallback/seq_lock_wide.rs":"e8f22b7bda9eb7b55c4c216dd78a2c4e3ca3ba8fda3385d1a55605e302d39f53","src/imp/fallback/utils.rs":"6d5926247f47004b74e6beb6cf1f2902f8fda628afd6cf4c739cf79a91122c46","src/imp/float.rs":"a981e4e197e55077635ce585a9b62a1ea20cbb4ab010749748adb8c22045c3a4","src/imp/interrupt/README.md":"14f2c84eea49ab7c0022b200750844851a116ac193dd8b9839971a89d94b2199","src/imp/interrupt/armv4t.rs":"d65849f9e0f4290e3fa9ba8c4f36d63616bf56d829ae89be60f6281d37e52928","src/imp/interrupt/armv6m.rs":"a594e9f7a01e788f55bec1636248ed0becc8abda769cbc62daa61ad5c8ce160e","src/imp/interrupt/avr.rs":"7a9b84a6a499fd46b4452d3e5c928178aa0be3443bee914569c143d4370a3747","src/imp/interrupt/mod.rs":"cc797fc537baaa061d90d74925321c53fd7e8412b02e5ee8ca6632c1eec8f734","src/imp/interrupt/msp430.rs":"d7af791094e05d3358f09d4c18b012380ac154f70970e89c5fc6cf7bdffda62e","src/imp/interrupt/riscv.rs":"236b3db723d55b0845f6fcdbf53cfaada6ff188fe1b543c2e6e1ba20dc66ab21","src/imp/interrupt/xtensa.rs":"8f9f462ad4196f8ec5247d871faeba3e468c417b698881bd62afe10a6c7e07b8","src/imp/mod.rs":"082f757d893d1c65e89834664531bcdb2e8083d51da5bcdfbebbe78a7861ef3f","src/imp/msp430.rs":"f812cfd6fbc3cc11b383e35924ed17e99da284fae1cb331c8e83e787c88f5ddc","src/imp/riscv.rs":"57352aa3f2f788bafaf968aed388c8c3e5b90c06cd211a0e33476ce7e0604831","src/imp/x86.rs":"d035c98653f93d08cf7c05fada6d631edbad06571a31b03eb3f0d5d7c0bf1cf3","src/lib.rs":"cb7532f2276b9928d67bd2e32bd113abff5ac7e1cd372ed95e029d2564f97d70","src/tests/helper.rs":"f95989458bb45223ab2facfa5bb6f8177114d2ff018538e77c2b4c5eaf66a5f5","src/tests/mod.rs":"ba5b6bda9fc5d5e637a5496fae0ef7b830ef86c0d180d4c14c1137fa0567f8fe","src/utils.rs":"e1c90e4648ad8d9390ada22ce871ca3b9df3e5fd52e7573f08a07bb0631c7420","version.rs":"07fb421f30f7be7788f85223bca1e38074e3ec8d0415abc8566f3ef9e3933f88"},"package":"280dc24453071f1b63954171985a0b0d30058d287960968b9b2aca264c8d4ee6"}
\ No newline at end of file
+{"files":{".cargo_vcs_info.json":"482f6b3cc06cfb0c7c9bfa8c260c040a02d762fa4603872cc844a975ab24a8f0","CHANGELOG.md":"b2562dcfd04a5c5907eb348296fa1ca57b7c7420b615c4384d1ee6b024741988","Cargo.lock":"4eea6ec59f568f532cac8e3e71029e24d2fa514d668dd4f86da5168c497b8cf6","Cargo.toml":"3366d6ebb1bada583bfdad414effb4776e4d090e22fd5bbd61c5b8a3560fa17b","Cargo.toml.orig":"ec7bbbc631719bf79a109603f496b852c636062776df74235987d74a825dbbcd","LICENSE-APACHE":"0d542e0c8804e39aa7f37eb00da5a762149dc682d7829451287e11b938e94594","LICENSE-MIT":"23f18e03dc49df91622fe2a76176497404e46ced8a715d9d2b67a7446571cca3","README.md":"f1c6a56dd42e2e1d02f5d7fef3d77dfca493db10d2409edaee7d48cdc1463b45","build.rs":"4bd0a08a8377b9a468a97a98197db80312a802221b1d7421d6efb40e9e447e2a","matrix-old.json":"7e9c4478ef365531a53834130a53da5b9864e786563682b1b9fcf4dd428dcaf3","src/cfgs.rs":"2e7bdfd6ba48957a8760c0cec389c91917bd5965abaa04db32f916e6e75e8aae","src/gen/build.rs":"4ad2fa43f8f725631e9754338f39b80f64a2e61facdb3bd735efc215b95a6597","src/gen/utils.rs":"49dd2efed43772f91e440523e39d7f9974c1a3305eabde4bccf4044a285dd589","src/imp/atomic128/README.md":"240d085f57eff9e5c92cb87498e0d1bb260a4f0fe41cd64eb86cb96e1d686c96","src/imp/atomic128/aarch64.rs":"c6ecab7ffa5c9609ceac0308e12c6ead732a3e246d75f726b5125879090e721e","src/imp/atomic128/intrinsics.rs":"d268e4aa067b25019fd6329d18f0604210d96af3ee6faedb7ec40d8748646d89","src/imp/atomic128/macros.rs":"31f63dd5f6523fdc80d752afdd7b13e90394e7210b8f14bfa38a4feab0822cd5","src/imp/atomic128/mod.rs":"4d12d39ed3c0e5ab46724ac9536598ccc0028147d258b739ed27212c4e4d5e7e","src/imp/atomic128/powerpc64.rs":"3434fc2bd7fa1d4c93205df7c88aea26934bdcbf867282e99cdbb9d1e09b8806","src/imp/atomic128/riscv64.rs":"6f7c2425c9ddaf5c349039b1d341d5f19c1265705fe9df53a677c81dc91bcfb5","src/imp/atomic128/s390x.rs":"1615997158bdad3bf7d16cafe221c8c0885688fa5e074352c31cad242ac98b0a","src/imp/atomic128/x86_64.rs":"dd5e5e05cc5fff506f99b9ece813cf3f113f618a173e2e94249dce077d42ea93","src/imp/atomic64/README.md":"56ef5c1f1750515a6a8a94b9347fd42faa889cddf5a9f77cd7d650008a4a8b7c","src/imp/atomic64/arm_linux.rs":"adfc11193d412e1c89c7ecdefbee81c5278878279a49c461ab0329c6da7522ac","src/imp/atomic64/macros.rs":"5b6f0bfe40064885926cfd57a336f15da92f6964864c50d339d1bb9903802a9d","src/imp/atomic64/mod.rs":"2977468d184134a068179eb8361d73500865b8d1476b87590099d6b503db30e5","src/imp/atomic64/riscv32.rs":"dfc4cf24ea2e02f4838326ccdcdbab81632ef146f8d49f9cc0fcc85ec838962a","src/imp/avr.rs":"fa50cc025a934377ecb7312fe78c519744809c14f1e1d00f73d8613ffc125df8","src/imp/core_atomic.rs":"e17dc0c7498493dea2ee2800b221e23f1d81403dfc4b308c9eff83efe9e0436c","src/imp/detect/README.md":"68db4ae18738636927628c41de5412e784a026f13529dda9cfe4f8407fd5f89c","src/imp/detect/aarch64_aa64reg.rs":"7d1564035c3a5d8ec177ee255860526c7adf5759db4d6380330a490c77d4d494","src/imp/detect/aarch64_apple.rs":"72d77305801b597b3cd30d1204666169aeffa551dfe88efe3f169e10804aaf25","src/imp/detect/aarch64_fuchsia.rs":"cff2f3808bcf30f2b11c7010a8514cc0bda766615a203da87262fec9d918102c","src/imp/detect/aarch64_illumos.rs":"edc9bf24fc43969d36b2d4aaaf45a622d7699c9c06a7cc716c5cbb94b57f10dc","src/imp/detect/aarch64_windows.rs":"4cdf3df6539205800919a6bd8c17eb238b571183bb853a7a4600ab7a7ed7dbe7","src/imp/detect/auxv.rs":"a58e6da2e71c8544f01e3f026591ee2807bf645fb73e2e97911d32bb56d24da7","src/imp/detect/common.rs":"5a9a16421edb7df226d3ab51a20fe6f8cfca1331fc92613583cd0b1e44b19ecb","src/imp/detect/powerpc64_aix.rs":"34f3694a359bd50f18633817528bcffb45bc42db8405a9ab49cefb25fb4af1ee","src/imp/detect/riscv_linux.rs":"aa10f712f6e27cb919cbff03987623f3f21d898fbccadbfc94fffd1f42e5bf80","src/imp/detect/x86_64.rs":"c2849317ab717a5e97b6a72ac048b0139a964c3e85162d7230458de97a7e644b","src/imp/fallback/mod.rs":"e63d984025ae98941e1c2403b2e5e428d4b5aa67a19c465e788aac2a7d18c77b","src/imp/fallback/outline_atomics.rs":"041d872affc4ad0b6b452f695a1373a1f76fe75d1b2377c24c3c0d199d89e383","src/imp/fallback/seq_lock.rs":"71aae56446d9e656037f5b13cc34a15f2ff061db7d4ea2b0a40ff721956dd138","src/imp/fallback/seq_lock_wide.rs":"2b6646e2f737b9253d8c9d143dc0dabbe3d675b9b973a26162b9bc5ab9fb472b","src/imp/fallback/utils.rs":"86f0d521455069988ea33453b1ab9fcf892ea62134686656f4e9875305ba9fdc","src/imp/float/aarch64.rs":"7dfa52d232470a8a0573972d9dde874f5bf03d4d7532174d851010b3686392a7","src/imp/float/int.rs":"8801b336650a039bc6e056857c0f65c2f148a0774c83baa9bd9bf415618107e5","src/imp/float/mod.rs":"2c722182baee05df65f9f6e2b4de5d5474a3b2bf0c24803a65a771de78b76f95","src/imp/interrupt/README.md":"1f4ed0b78b42832800bc8e4db98576f17ef51e8e84b07375fa3e076addbf83e0","src/imp/interrupt/armv4t.rs":"6fb9e7d01f7fb483bc6bec07916fdcba8563b605b53bd87c5310662da04e55a4","src/imp/interrupt/armv6m.rs":"364a38f18ffb2b3cd836700dff6a390d9542f0f2a3eda07bc63bc8868dafaad1","src/imp/interrupt/avr.rs":"17896e2d3e37a03ddc90e884ba09b3e241a8e56ef611fe9790b799c19494b75f","src/imp/interrupt/mod.rs":"27593db509704456fc853f8c6fe28e5dd055b31ddd33a01e4fd6d88d01df4919","src/imp/interrupt/msp430.rs":"bb752aad62bbbddd3b190702ada84bb562e930752eae7de0c96ca0abc620fbe8","src/imp/interrupt/riscv.rs":"f27fe895d55da2ac097fcd1db64cc455a88d0f0d46668cf4ebb61155f8334281","src/imp/interrupt/xtensa.rs":"69e45c6e6a82e06a827967d34268be8c6545138112b46fd7a6610a68f51ba113","src/imp/mod.rs":"2c269d5051f136669cce386cefbe1be90a79e6731f9c42de83091813581a8bea","src/imp/msp430.rs":"16635485eeca1c1c4b866253eefc1f39fa6121c84c16ec389b904c03fa9cdb55","src/imp/riscv.rs":"6a7b7332800141a4da74e1b241e6c04e519b64e4d69ff9b9ef0f9b6a893cb6b0","src/imp/x86.rs":"97c0e1743d7ba7a3dbabd9f49c38ffc71447863ff9c91db53963e003d2b0adf8","src/lib.rs":"5fc1d48545cd2fcd1cba81898657de683f679c6a96b1bd8aea284c19265f167b","src/rustdoc.css":"fb503ec0cf2eb0dacbbdbea63ffb66d538637ddb0a6617cad69906bf58831d7f","src/tests/helper.rs":"08b56889f8fb78efcbd58bb835b66f05dc15672de8cd216d05a12d6a48460563","src/tests/mod.rs":"c1a316f83894a71f1fc870db04d5401774162a277fd7d0b33f955ce35fc05399","src/utils.rs":"3687375efdb580dee21373a9c427030757f640f23b140d4a8d8b47a4085104fa","version.rs":"cc5733c63aeabd7f37c7ae0c47f0be6a7d7a843f911ca181a4d41d9d4fd4c8f0"},"package":"c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49"}
\ No newline at end of file
### external/vendor/portable-atomic/.cargo_vcs_info.json
@@ -1,6 +1,6 @@
{
"git": {
- "sha1": "b70300628520f2cab40798e58aa48af690bc94e7"
+ "sha1": "1639861c663868f83a38dd7798aae30c5bb21c62"
},
"path_in_vcs": ""
}
\ No newline at end of file
### external/vendor/portable-atomic/CHANGELOG.md
@@ -12,6 +12,71 @@ Note: In this file, do not use the hard wrap in the middle of a sentence for com
## [Unreleased]
+## [1.13.1] - 2026-01-31
+
+- Update to stabilized [PowerPC64](https://github.com/rust-lang/rust/pull/147996) inline assembly. ([92b02f8a](https://github.com/taiki-e/portable-atomic/commit/92b02f8a279327a1780cbe127d9effb2baae9b2f))
+
+- Work around [rustc_codegen_gcc bugs on x86_64](https://github.com/rust-lang/rustc_codegen_gcc/issues/821#issuecomment-3793567607). ([ae4c501](https://github.com/taiki-e/portable-atomic/commit/ae4c501aec84a3537fe35ec57ceae94b3a05ade0))
+
+- Optimize x86_64 128-bit atomics. ([a9d61eb](https://github.com/taiki-e/portable-atomic/commit/a9d61ebf8d7f466286a71a17f7d9063fcf07fce0), [90a17ca4](https://github.com/taiki-e/portable-atomic/commit/90a17ca40a8ff433d767c3b56264fb02ccdd71e1))
+
+- Improve compile-time detection of RISC-V target features. ([535fced](https://github.com/taiki-e/portable-atomic/commit/535fced071ed095ee4d35b440ba55a0e2f533d80))
+
+- Enable [release immutability](https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/immutable-releases).
+
+## [1.13.0] - 2025-12-27
+
+- Add `unsafe-assume-privileged` feature / `portable_atomic_unsafe_assume_privileged` cfg for safer lock-based fallback on multi-core privileged environments. ([b084ee1](https://github.com/taiki-e/portable-atomic/commit/b084ee1b6cba3e9d7158a6c9e5450e1fc8bbde36))
+
+- Support `unsafe-assume-single-core`/`unsafe-assume-privileged` on all 32-bit Arm targets. Previously it was only M-profile and pre-v6 targets. ([7e07f5e](https://github.com/taiki-e/portable-atomic/commit/7e07f5e2bc8ad74287830522c02f960a8c8da59e))
+
+- Make `AtomicPtr::fetch_*` strict-provenance compatible on all environments. Previously it was only strict-provenance compatible on `cfg(miri)` and otherwise permissive-provenance compatible. ([4306943](https://github.com/taiki-e/portable-atomic/commit/4306943fb09af3a4f763f1f8ff257fe752c7b7e3))
+
+- Ensure sequential consistency in lock-based fallback when SeqCst is used. ([7e80742](https://github.com/taiki-e/portable-atomic/commit/7e80742eeed9fc4d1aa15455b862d70194f6f1bf))
+
+- Support compile-time detection for x86_64 VMOVDQA. ([f7bb1aa](https://github.com/taiki-e/portable-atomic/commit/f7bb1aa246df0e13fa02fb707f8462d8dfe6b7e9))
+
+- Improve compile-time detection of s390x miscellaneous-extensions-3. ([11045fe](https://github.com/taiki-e/portable-atomic/commit/11045fe513689a842e393324c05f2c5f169b59d4))
+
+- Optimize AVR 8-bit swap when RMW instructions available. ([8cedb34](https://github.com/taiki-e/portable-atomic/commit/8cedb34a0b9f2ca1680d893a58020ba1e5d0a87b))
+
+- Optimize interrupt restore on RISC-V. ([9b97a2a](https://github.com/taiki-e/portable-atomic/commit/9b97a2a18142c9a8d21f03de7fff30caea89d51e))
+
+## [1.12.0] - 2025-12-19
+
+- Fix build error on no-std pre-v6 Arm targets due to the [recent upstream change](https://github.com/rust-lang/rust/pull/149241). ([83f6f3e](https://github.com/taiki-e/portable-atomic/commit/83f6f3e4957833af6dd1bae054da1e8d51501a76))
+
+- Support `unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg for targets with CAS. ([38e9572](https://github.com/taiki-e/portable-atomic/commit/38e95722dde98e7a9b59d2acbff968450a0b09ea))
+
+- Improve compile-time detection of s390x target feature. ([5ae0ef5](https://github.com/taiki-e/portable-atomic/commit/5ae0ef5ed7f9a0c9efe9e628ba2fbc5876487219))
+
+- Documentation improvements. ([c84f720](https://github.com/taiki-e/portable-atomic/commit/c84f7203ae6e39a5c9966748123b95b90e8a127a))
+
+## [1.11.1] - 2025-06-06
+
+- Fix build error when building aarch64/arm64ec/powerpc64/s390x targets for Miri or ThreadSanitizer since nightly-2025-05-31.
+
+- aarch64: Optimize atomic floats when FEAT_LSFE is enabled. ([#201](https://github.com/taiki-e/portable-atomic/pull/201))
+
+- Improve compile-time detection of RISC-V Zacas extension. ([b7634e2](https://github.com/taiki-e/portable-atomic/commit/b7634e2cd808ea118266d12f99fd8877a92e3d31))
+
+- Improve run-time detection on linux-musl. ([7fdad7f](https://github.com/taiki-e/portable-atomic/commit/7fdad7f7dd32e32ece7bd0eaf565db657b3406bb))
+
+- Optimize interrupt restore on thumbv6m. ([dd2004a](https://github.com/taiki-e/portable-atomic/commit/dd2004aaa14d034b0db652eb9939b780d0d8221f))
+
+## [1.11.0] - 2025-02-24
+
+- Work around [nightly-2025-02-24 rustc regression causing "cannot use value of type `*mut T` for inline assembly" error](https://github.com/rust-lang/rust/issues/137512) on RISC-V without A extension, MSP430, and pre-v6 no-std Arm targets. ([eeb0235](https://github.com/taiki-e/portable-atomic/commit/eeb0235b9fda4c28a56ee5a9ffe0d7fb884a50ab))
+
+- Support `AtomicF16` and `AtomicF128` for [unstable `f16` and `f128`](https://github.com/rust-lang/rust/issues/116909) under unstable cfgs. ([#200](https://github.com/taiki-e/portable-atomic/pull/200))
+
+- RISC-V Zacas extension support is no longer experimental. ([#206](https://github.com/taiki-e/portable-atomic/pull/206))
+
+- Improve support of run-time detection and outline-atomics:
+ - riscv: Enable run-time detection of Zacas extension by default on Linux/Android. ([#207](https://github.com/taiki-e/portable-atomic/pull/207))
+ - aarch64: Support run-time detection of FEAT_LRCPC3/FEAT_LSE128 on FreeBSD. ([6a5075d](https://github.com/taiki-e/portable-atomic/commit/6a5075d43543875cf38d6114f2951047e2e64f1a))
+ - powerpc64: Support run-time detection of quadword-atomics on AIX (currently disabled by default because detection support for AIX is experimental). ([#102](https://github.com/taiki-e/portable-atomic/pull/102))
+
## [1.10.0] - 2024-11-23
- Update to stabilized [s390x](https://github.com/rust-lang/rust/pull/131258) and [Arm64EC](https://github.com/rust-lang/rust/pull/131781) inline assembly. ([97645c1](https://github.com/taiki-e/portable-atomic/commit/97645c1b2b938249f16eacb0fe696d4c7bb96754), [e1d1a97](https://github.com/taiki-e/portable-atomic/commit/e1d1a97cd1ab4bd04b45962c44ca1e9f0f9e1456))
@@ -509,7 +574,12 @@ The latest version of portable-atomic is 1.x. This release makes portable-atomic
Initial release
-[Unreleased]: https://github.com/taiki-e/portable-atomic/compare/v1.10.0...HEAD
+[Unreleased]: https://github.com/taiki-e/portable-atomic/compare/v1.13.1...HEAD
+[1.13.1]: https://github.com/taiki-e/portable-atomic/compare/v1.13.0...v1.13.1
+[1.13.0]: https://github.com/taiki-e/portable-atomic/compare/v1.12.0...v1.13.0
+[1.12.0]: https://github.com/taiki-e/portable-atomic/compare/v1.11.1...v1.12.0
+[1.11.1]: https://github.com/taiki-e/portable-atomic/compare/v1.11.0...v1.11.1
+[1.11.0]: https://github.com/taiki-e/portable-atomic/compare/v1.10.0...v1.11.0
[1.10.0]: https://github.com/taiki-e/portable-atomic/compare/v1.9.0...v1.10.0
[1.9.0]: https://github.com/taiki-e/portable-atomic/compare/v1.8.0...v1.9.0
[1.8.0]: https://github.com/taiki-e/portable-atomic/compare/v1.7.0...v1.8.0
### external/vendor/portable-atomic/Cargo.lock
@@ -0,0 +1,250 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+[[package]]
+name = "build-context"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "cc"
+version = "1.2.55"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "find-msvc-tools 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)",
+ "jobserver 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)",
+ "libc 0.2.163 (registry+https://github.com/rust-lang/crates.io-index)",
+ "shlex 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "cfg-if"
+version = "1.0.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "crabgrind"
+version = "0.1.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "cc 1.2.55 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "critical-section"
+version = "1.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "crossbeam-utils"
+version = "0.8.16"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "cfg-if 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "fastrand"
+version = "2.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "find-msvc-tools"
+version = "0.1.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "getrandom"
+version = "0.2.17"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "cfg-if 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)",
+ "libc 0.2.163 (registry+https://github.com/rust-lang/crates.io-index)",
+ "wasi 0.11.1+wasi-snapshot-preview1 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "jobserver"
+version = "0.1.32"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "libc 0.2.163 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "libc"
+version = "0.2.163"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "paste"
+version = "1.0.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "portable-atomic"
+version = "1.13.1"
+dependencies = [
+ "build-context 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
+ "crabgrind 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)",
+ "critical-section 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "crossbeam-utils 0.8.16 (registry+https://github.com/rust-lang/crates.io-index)",
+ "fastrand 2.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "libc 0.2.163 (registry+https://github.com/rust-lang/crates.io-index)",
+ "paste 1.0.15 (registry+https://github.com/rust-lang/crates.io-index)",
+ "quickcheck 1.0.3 (registry+https://github.com/rust-lang/crates.io-index)",
+ "rustversion 1.0.22 (registry+https://github.com/rust-lang/crates.io-index)",
+ "serde 1.0.228 (registry+https://github.com/rust-lang/crates.io-index)",
+ "sptr 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)",
+ "static_assertions 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "windows-sys 0.61.2 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "proc-macro2"
+version = "1.0.106"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "unicode-ident 1.0.22 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "quickcheck"
+version = "1.0.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "rand 0.8.5 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "quote"
+version = "1.0.44"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "proc-macro2 1.0.106 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "rand"
+version = "0.8.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "rand_core 0.6.4 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "rand_core"
+version = "0.6.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "getrandom 0.2.17 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "rustversion"
+version = "1.0.22"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "serde"
+version = "1.0.228"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "serde_core 1.0.228 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "serde_core"
+version = "1.0.228"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "serde_derive 1.0.228 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "serde_derive"
+version = "1.0.228"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "proc-macro2 1.0.106 (registry+https://github.com/rust-lang/crates.io-index)",
+ "quote 1.0.44 (registry+https://github.com/rust-lang/crates.io-index)",
+ "syn 2.0.114 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "shlex"
+version = "1.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "sptr"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "static_assertions"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "syn"
+version = "2.0.114"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "proc-macro2 1.0.106 (registry+https://github.com/rust-lang/crates.io-index)",
+ "quote 1.0.44 (registry+https://github.com/rust-lang/crates.io-index)",
+ "unicode-ident 1.0.22 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "unicode-ident"
+version = "1.0.22"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "wasi"
+version = "0.11.1+wasi-snapshot-preview1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "windows-link"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
+[[package]]
+name = "windows-sys"
+version = "0.61.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "windows-link 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[metadata]
+"checksum build-context 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "86610cb1e9d45d65a31b574f9d69de003a76b6bb0b7d882396a5153fc547c935"
+"checksum cc 1.2.55 (registry+https://github.com/rust-lang/crates.io-index)" = "47b26a0954ae34af09b50f0de26458fa95369a0d478d8236d3f93082b219bd29"
+"checksum cfg-if 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)" = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
+"checksum crabgrind 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)" = "f9e452c480754e99194a6574d5588f3bb892e1974c15439e5af8df4a385b9a8a"
+"checksum critical-section 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b"
+"checksum crossbeam-utils 0.8.16 (registry+https://github.com/rust-lang/crates.io-index)" = "5a22b2d63d4d1dc0b7f1b6b2747dd0088008a9be28b6ddf0b1e7d335e3037294"
+"checksum fastrand 2.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be"
+"checksum find-msvc-tools 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)" = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
+"checksum getrandom 0.2.17 (registry+https://github.com/rust-lang/crates.io-index)" = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
+"checksum jobserver 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)" = "48d1dbcbbeb6a7fec7e059840aa538bd62aaccf972c7346c4d9d2059312853d0"
+"checksum libc 0.2.163 (registry+https://github.com/rust-lang/crates.io-index)" = "1fdaeca4cf44ed4ac623e86ef41f056e848dbeab7ec043ecb7326ba300b36fd0"
+"checksum paste 1.0.15 (registry+https://github.com/rust-lang/crates.io-index)" = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a"
+"checksum proc-macro2 1.0.106 (registry+https://github.com/rust-lang/crates.io-index)" = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
+"checksum quickcheck 1.0.3 (registry+https://github.com/rust-lang/crates.io-index)" = "588f6378e4dd99458b60ec275b4477add41ce4fa9f64dcba6f15adccb19b50d6"
+"checksum quote 1.0.44 (registry+https://github.com/rust-lang/crates.io-index)" = "21b2ebcf727b7760c461f091f9f0f539b77b8e87f2fd88131e7f1b433b3cece4"
+"checksum rand 0.8.5 (registry+https://github.com/rust-lang/crates.io-index)" = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404"
+"checksum rand_core 0.6.4 (registry+https://github.com/rust-lang/crates.io-index)" = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
+"checksum rustversion 1.0.22 (registry+https://github.com/rust-lang/crates.io-index)" = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
+"checksum serde 1.0.228 (registry+https://github.com/rust-lang/crates.io-index)" = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
+"checksum serde_core 1.0.228 (registry+https://github.com/rust-lang/crates.io-index)" = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
+"checksum serde_derive 1.0.228 (registry+https://github.com/rust-lang/crates.io-index)" = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
+"checksum shlex 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
+"checksum sptr 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)" = "3b9b39299b249ad65f3b7e96443bad61c02ca5cd3589f46cb6d610a0fd6c0d6a"
+"checksum static_assertions 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
+"checksum syn 2.0.114 (registry+https://github.com/rust-lang/crates.io-index)" = "d4d107df263a3013ef9b1879b0df87d706ff80f65a86ea879bd9c31f9b307c2a"
+"checksum unicode-ident 1.0.22 (registry+https://github.com/rust-lang/crates.io-index)" = "9312f7c4f6ff9069b165498234ce8be658059c6728633667c526e27dc2cf1df5"
+"checksum wasi 0.11.1+wasi-snapshot-preview1 (registry+https://github.com/rust-lang/crates.io-index)" = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
+"checksum windows-link 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
+"checksum windows-sys 0.61.2 (registry+https://github.com/rust-lang/crates.io-index)" = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
### external/vendor/portable-atomic/Cargo.toml
@@ -13,14 +13,15 @@
edition = "2018"
rust-version = "1.34"
name = "portable-atomic"
-version = "1.10.0"
+version = "1.13.1"
build = "build.rs"
exclude = [
"/.*",
"/tools",
"/target-specs",
"/DEVELOPMENT.md",
]
+autolib = false
autobins = false
autoexamples = false
autotests = false
@@ -40,18 +41,38 @@ categories = [
license = "Apache-2.0 OR MIT"
repository = "https://github.com/taiki-e/portable-atomic"
-[package.metadata.cargo_check_external_types]
-allowed_external_types = ["serde::*"]
-
[package.metadata.docs.rs]
features = [
"float",
"std",
"serde",
"critical-section",
]
+rustdoc-args = [
+ "--extend-css",
+ "src/rustdoc.css",
+ "--cfg",
+ "portable_atomic_unstable_f16",
+ "--cfg",
+ "portable_atomic_unstable_f128",
+]
targets = ["x86_64-unknown-linux-gnu"]
+[package.metadata.cargo_check_external_types]
+allowed_external_types = ["serde_core::*"]
+
+[features]
+default = ["fallback"]
+disable-fiq = []
+fallback = []
+float = []
+force-amo = []
+require-cas = []
+s-mode = []
+std = []
+unsafe-assume-privileged = []
+unsafe-assume-single-core = []
+
[lib]
name = "portable_atomic"
path = "src/lib.rs"
@@ -78,28 +99,27 @@ version = "2"
[dev-dependencies.paste]
version = "1"
+[dev-dependencies.quickcheck]
+version = "1"
+default-features = false
+
+[dev-dependencies.rustversion]
+version = "1"
+
[dev-dependencies.sptr]
version = "0.3"
[dev-dependencies.static_assertions]
version = "1"
-[features]
-default = ["fallback"]
-disable-fiq = []
-fallback = []
-float = []
-force-amo = []
-require-cas = []
-s-mode = []
-std = []
-unsafe-assume-single-core = []
-
[target."cfg(unix)".dev-dependencies.libc]
version = "=0.2.163"
+[target."cfg(valgrind)".dev-dependencies.crabgrind]
+version = "0.1"
+
[target."cfg(windows)".dev-dependencies.windows-sys]
-version = "0.59"
+version = "0.61"
features = [
"Win32_Foundation",
"Win32_System_Threading",
@@ -115,6 +135,7 @@ pedantic = "warn"
trailing_empty_array = "warn"
transmute_undefined_repr = "warn"
undocumented_unsafe_blocks = "warn"
+unused_trait_names = "warn"
[lints.clippy.bool_assert_comparison]
level = "allow"
@@ -219,20 +240,29 @@ improper_ctypes_definitions = "warn"
non_ascii_idents = "warn"
rust_2018_idioms = "warn"
single_use_lifetimes = "warn"
+unnameable_types = "warn"
unreachable_pub = "warn"
[lints.rust.unexpected_cfgs]
level = "warn"
priority = 0
check-cfg = [
'cfg(target_arch,values("xtensa"))',
+ 'cfg(target_arch,values("amdgpu"))',
+ 'cfg(target_arch,values("loongarch32"))',
+ 'cfg(target_os,values("trusty"))',
'cfg(target_os,values("psx"))',
'cfg(target_env,values("psx"))',
'cfg(target_feature,values("lse2","lse128","rcpc3"))',
'cfg(target_feature,values("quadword-atomics"))',
'cfg(target_feature,values("zaamo","zabha"))',
+ 'cfg(target_feature,values("zacas"))',
+ 'cfg(target_feature,values("miscellaneous-extensions-3"))',
'cfg(target_pointer_width,values("128"))',
- "cfg(portable_atomic_test_outline_atomics_detect_false,qemu,valgrind)",
- "cfg(portable_atomic_no_outline_atomics,portable_atomic_outline_atomics)",
+ "cfg(portable_atomic_no_outline_atomics,portable_atomic_outline_atomics,portable_atomic_unstable_f16,portable_atomic_unstable_f128)",
"cfg(portable_atomic_unstable_coerce_unsized)",
+ "cfg(portable_atomic_test_detect_false,portable_atomic_test_no_std_static_assert_ffi,qemu,valgrind)",
]
+
+[profile.release]
+debug = 2
### external/vendor/portable-atomic/Cargo.toml.orig
@@ -1,8 +1,8 @@
[package]
name = "portable-atomic"
-version = "1.10.0" #publish:version
+version = "1.13.1" #publish:version
edition = "2018"
-rust-version = "1.34"
+rust-version = "1.34" # For Atomic{I,U}{8,16,32,64}
license = "Apache-2.0 OR MIT"
repository = "https://github.com/taiki-e/portable-atomic"
keywords = ["atomic"]
@@ -17,91 +17,68 @@ Portable atomic types including support for 128-bit atomics, atomic float, etc.
# - env.TEST_FEATURES in .github/workflows/ci.yml.
# - test_features list in tools/build.sh and tools/test.sh.
features = ["float", "std", "serde", "critical-section"]
+rustdoc-args = ["--extend-css", "src/rustdoc.css", "--cfg", "portable_atomic_unstable_f16", "--cfg", "portable_atomic_unstable_f128"]
targets = ["x86_64-unknown-linux-gnu"]
[package.metadata.cargo_check_external_types]
# The following are external types that are allowed to be exposed in our public API.
allowed_external_types = [
- "serde::*",
+ "serde_core::*",
]
[lib]
doc-scrape-examples = false
+# Please read the documentation before using optional features:
+# https://github.com/taiki-e/portable-atomic#optional-features
[features]
default = ["fallback"]
-
-# (enabled by default) Enable fallback implementations.
-#
-# Disabling this allows only atomic types for which the platform natively supports atomic operations.
fallback = []
-
-# Provide `AtomicF{32,64}`.
-#
-# See documentation for more: https://github.com/taiki-e/portable-atomic#optional-features-float
float = []
-
-# Use `std`.
std = []
-
-# Emit compile error if atomic CAS is not available.
-#
-# See documentation for more: https://github.com/taiki-e/portable-atomic#optional-features-require-cas
require-cas = []
-
-# Assume the target is single core, to enable implementations based on disabling interrupts.
-# IMPORTANT: This feature is unsafe. See the documentation for the safety contract:
-# https://github.com/taiki-e/portable-atomic#optional-features-unsafe-assume-single-core
unsafe-assume-single-core = []
-
-# The following are sub-features of the unsafe-assume-single-core feature and if enabled without
-# the unsafe-assume-single-core feature will result in a compile error.
-# There is no explicit "unsafe-" prefix because the user has already opted in to "unsafe" by
-# enabling the unsafe-assume-single-core feature, but misuse of these features is also usually
-# considered unsound.
-# See the interrupt module's readme for more: https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/interrupt/README.md
-
-# For RISC-V targets, generate code for S mode to disable interrupts.
+unsafe-assume-privileged = []
s-mode = []
-# For RISC-V targets, use AMO instructions even if A-extension is disabled.
-# This feature requires Rust 1.72+.
force-amo = []
-# For Arm targets, also disable FIQs when disabling interrupts.
disable-fiq = []
# Note: serde and critical-section are public dependencies.
[dependencies]
-# Implements serde::{Serialize,Deserialize} for atomic types.
-#
-# See documentation for more: https://github.com/taiki-e/portable-atomic#optional-features-serde
+# Please read the documentation before using optional features:
+# https://github.com/taiki-e/portable-atomic#optional-features
serde = { version = "1.0.60", optional = true, default-features = false }
-
-# Use `critical-section`.
-#
-# See documentation for more: https://github.com/taiki-e/portable-atomic#optional-features-critical-section
critical-section = { version = "1", optional = true }
-[dev-dependencies]
-test-helper = { git = "https://github.com/taiki-e/test-helper.git", branch = "main", features = ["std", "sys", "cpuinfo", "critical-section"] }
+# [target.'cfg(portable_atomic_test_no_std_static_assert_ffi)'.dependencies] #build:static_assert_ffi
+# test-helper = { features = ["sys"], git = "https://github.com/taiki-e/test-helper.git", rev = "e2e8e37" } #build:static_assert_ffi
+# [target.'cfg(all(portable_atomic_test_no_std_static_assert_ffi, target_os = "aix"))'.dependencies] #build:static_assert_ffi
+# libc = "=0.2.163" #build:static_assert_ffi
+[dev-dependencies]
build-context = "0.1"
crossbeam-utils = "=0.8.16" # The latest crossbeam-utils requires Rust 1.60
fastrand = "2"
paste = "1"
-quickcheck = { default-features = false, git = "https://github.com/taiki-e/quickcheck.git", branch = "dev" } # https://github.com/BurntSushi/quickcheck/pull/304 + https://github.com/BurntSushi/quickcheck/pull/282 + https://github.com/BurntSushi/quickcheck/pull/296 + lower MSRV
-serde_test = { git = "https://github.com/taiki-e/serde_test.git", branch = "dev" } # support {i,u}128
+quickcheck = { version = "1", default-features = false, git = "https://github.com/taiki-e/quickcheck.git", rev = "83b1d59" } # https://github.com/BurntSushi/quickcheck/pull/304 + https://github.com/BurntSushi/quickcheck/pull/282 + https://github.com/BurntSushi/quickcheck/pull/296 + f16/f128 support + lower MSRV
+rustversion = "1"
+serde_test = { git = "https://github.com/taiki-e/serde_test.git", rev = "df513c5" } # support {i,u}128
sptr = "0.3"
static_assertions = "1"
+test-helper = { features = ["sys", "critical-section-std"], git = "https://github.com/taiki-e/test-helper.git", rev = "e2e8e37" }
[target.'cfg(unix)'.dev-dependencies]
libc = "=0.2.163" # newer libc requires Rust 1.63
[target.'cfg(windows)'.dev-dependencies]
-windows-sys = { version = "0.59", features = [
+windows-sys = { version = "0.61", features = [
"Win32_Foundation",
- "Win32_System_Threading",
+ "Win32_System_Threading", # IsProcessorFeaturePresent
] }
+[target.'cfg(valgrind)'.dev-dependencies]
+crabgrind = "0.1"
+
[lints]
workspace = true
@@ -113,7 +90,7 @@ members = [
]
# This table is shared by projects under github.com/taiki-e.
-# It is not intended for manual editing.
+# Expect for unexpected_cfgs.check-cfg, it is not intended for manual editing.
[workspace.lints.rust]
deprecated_safe = "warn"
improper_ctypes = "warn"
@@ -123,22 +100,28 @@ rust_2018_idioms = "warn"
single_use_lifetimes = "warn"
unexpected_cfgs = { level = "warn", check-cfg = [
'cfg(target_arch,values("xtensa"))', # 1.81+ https://github.com/rust-lang/rust/pull/125141
+ 'cfg(target_arch,values("amdgpu"))', # 1.86+ https://github.com/rust-lang/rust/pull/134740
+ 'cfg(target_arch,values("loongarch32"))', # 1.89+ https://github.com/rust-lang/rust/pull/142053
+ 'cfg(target_os,values("trusty"))', # 1.82+ https://github.com/rust-lang/rust/pull/129490
'cfg(target_os,values("psx"))', # 1.84+ https://github.com/rust-lang/rust/pull/131168
'cfg(target_env,values("psx"))', # pre-1.84 https://github.com/rust-lang/rust/pull/131168
'cfg(target_feature,values("lse2","lse128","rcpc3"))', # 1.82+ https://github.com/rust-lang/rust/pull/128192
'cfg(target_feature,values("quadword-atomics"))', # 1.83+ https://github.com/rust-lang/rust/pull/130873
'cfg(target_feature,values("zaamo","zabha"))', # 1.83+ https://github.com/rust-lang/rust/pull/130877
+ 'cfg(target_feature,values("zacas"))', # 1.87+ https://github.com/rust-lang/rust/pull/137417
+ 'cfg(target_feature,values("miscellaneous-extensions-3"))', # 1.89+ https://github.com/rust-lang/rust/pull/141250
'cfg(target_pointer_width,values("128"))',
# Known custom cfgs, excluding those that may be set by build script.
- # Not public API.
- 'cfg(portable_atomic_test_outline_atomics_detect_false,qemu,valgrind)',
- # Public APIs, considered unstable unless documented in readme.
- 'cfg(portable_atomic_no_outline_atomics,portable_atomic_outline_atomics)',
+ # Public APIs, considered unstable unless documented as stable in readme.
+ 'cfg(portable_atomic_no_outline_atomics,portable_atomic_outline_atomics,portable_atomic_unstable_f16,portable_atomic_unstable_f128)',
# Public unstable API(s) - portable-atomic-util
'cfg(portable_atomic_unstable_coerce_unsized)',
+ # Not public API.
+ 'cfg(portable_atomic_test_detect_false,portable_atomic_test_no_std_static_assert_ffi,qemu,valgrind)',
] }
+unnameable_types = "warn"
unreachable_pub = "warn"
-# unsafe_op_in_unsafe_fn = "warn" # Set at crate-level instead since https://github.com/rust-lang/rust/pull/100081 is not available on MSRV
+# unsafe_op_in_unsafe_fn = "warn" # Set at crate-level instead since https://github.com/rust-lang/rust/pull/100081 merged in Rust 1.65 is not available on MSRV
[workspace.lints.clippy]
all = "warn" # Downgrade deny-by-default lints
pedantic = "warn"
@@ -149,6 +132,7 @@ inline_asm_x86_att_syntax = "warn"
trailing_empty_array = "warn"
transmute_undefined_repr = "warn"
undocumented_unsafe_blocks = "warn"
+unused_trait_names = "warn"
# Suppress buggy or noisy clippy lints
bool_assert_comparison = { level = "allow", priority = 1 }
borrow_as_ptr = { level = "allow", priority = 1 } # https://github.com/rust-lang/rust-clippy/issues/8286
@@ -174,3 +158,6 @@ too_many_arguments = { level = "allow", priority = 1 }
too_many_lines = { level = "allow", priority = 1 }
type_complexity = { level = "allow", priority = 1 }
unreadable_literal = { level = "allow", priority = 1 }
+
+[profile.release]
+debug = true
### external/vendor/portable-atomic/README.md
@@ -5,18 +5,18 @@
[](#license)
[](https://www.rust-lang.org)
[](https://github.com/taiki-e/portable-atomic/actions)
-[](https://cirrus-ci.com/github/taiki-e/portable-atomic)
-<!-- tidy:crate-doc:start -->
+<!-- tidy:sync-markdown-to-rustdoc:start:src/lib.rs -->
+
Portable atomic types including support for 128-bit atomics, atomic float, etc.
- Provide all atomic integer types (`Atomic{I,U}{8,16,32,64}`) for all targets that can use atomic CAS. (i.e., all targets that can use `std`, and most no-std targets)
- Provide `AtomicI128` and `AtomicU128`.
- Provide `AtomicF32` and `AtomicF64`. ([optional, requires the `float` feature](#optional-features-float))
+- Provide `AtomicF16` and `AtomicF128` for [unstable `f16` and `f128`](https://github.com/rust-lang/rust/issues/116909). ([optional, requires the `float` feature and unstable cfgs](#optional-features-float))
- Provide atomic load/store for targets where atomic is not available at all in the standard library. (RISC-V without A-extension, MSP430, AVR)
- Provide atomic CAS for targets where atomic CAS is not available in the standard library. (thumbv6m, pre-v6 Arm, RISC-V without A-extension, MSP430, AVR, Xtensa, etc.) (always enabled for MSP430 and AVR, [optional](#optional-features-critical-section) otherwise)
-- Provide stable equivalents of the standard library's atomic types' unstable APIs, such as [`AtomicPtr::fetch_*`](https://github.com/rust-lang/rust/issues/99108).
-- Make features that require newer compilers, such as [`fetch_{max,min}`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.fetch_max), [`fetch_update`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.fetch_update), [`as_ptr`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.as_ptr), [`from_ptr`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.from_ptr), [`AtomicBool::fetch_not`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicBool.html#method.fetch_not) and [stronger CAS failure ordering](https://github.com/rust-lang/rust/pull/98383) available on Rust 1.34+.
+- Make features that require newer compilers, such as [`fetch_{max,min}`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.fetch_max), [`fetch_update`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.fetch_update), [`as_ptr`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.as_ptr), [`from_ptr`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.from_ptr), [`AtomicBool::fetch_not`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicBool.html#method.fetch_not), [`AtomicPtr::fetch_*`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicPtr.html#method.fetch_and), and [stronger CAS failure ordering](https://github.com/rust-lang/rust/pull/98383) available on Rust 1.34+.
- Provide workaround for bugs in the standard library's atomic-related APIs, such as [rust-lang/rust#100650], `fence`/`compiler_fence` on MSP430 that cause LLVM error, etc.
<!-- TODO:
@@ -43,141 +43,184 @@ If you don't need them, disabling the default features may reduce code size and
portable-atomic = { version = "1", default-features = false }
```
-If your crate supports no-std environment and requires atomic CAS, enabling the `require-cas` feature will allow the `portable-atomic` to display a [helpful error message](https://github.com/taiki-e/portable-atomic/pull/100) to users on targets requiring additional action on the user side to provide atomic CAS.
+If your crate supports no-std environment and requires atomic CAS, enabling the `require-cas` feature will allow the portable-atomic to display a [helpful error message](https://github.com/taiki-e/portable-atomic/pull/100) to users on targets requiring additional action on the user side to provide atomic CAS.
```toml
[dependencies]
portable-atomic = { version = "1.3", default-features = false, features = ["require-cas"] }
```
+(Since 1.8, portable-atomic can display a [helpful error message](https://github.com/taiki-e/portable-atomic/pull/181) even without the `require-cas` feature when the rustc version is 1.78+. However, the `require-cas` feature also allows rejecting builds at an earlier stage, we recommend enabling it unless enabling it causes [problems](https://github.com/matklad/once_cell/pull/267).)
+
## 128-bit atomics support
-Native 128-bit atomic operations are available on x86_64 (Rust 1.59+), AArch64 (Rust 1.59+), riscv64 (Rust 1.59+), Arm64EC (Rust 1.84+), s390x (Rust 1.84+), and powerpc64 (nightly only), otherwise the fallback implementation is used.
+Native 128-bit atomic operations are available on x86_64 (Rust 1.59+), AArch64 (Rust 1.59+), riscv64 (Rust 1.59+), Arm64EC (Rust 1.84+), s390x (Rust 1.84+), and powerpc64 (Rust 1.95+), otherwise the fallback implementation is used.
-On x86_64, even if `cmpxchg16b` is not available at compile-time (note: `cmpxchg16b` target feature is enabled by default only on Apple and Windows (except Windows 7) targets), run-time detection checks whether `cmpxchg16b` is available. If `cmpxchg16b` is not available at either compile-time or run-time detection, the fallback implementation is used. See also [`portable_atomic_no_outline_atomics`](#optional-cfg-no-outline-atomics) cfg.
+On x86_64, even if `cmpxchg16b` is not available at compile-time (Note: `cmpxchg16b` target feature is enabled by default only on Apple, Windows (except Windows 7), and Fuchsia targets), run-time detection checks whether `cmpxchg16b` is available. If `cmpxchg16b` is not available at either compile-time or run-time detection, the fallback implementation is used. See also [`portable_atomic_no_outline_atomics`](#optional-cfg-no-outline-atomics) cfg.
They are usually implemented using inline assembly, and when using Miri or ThreadSanitizer that do not support inline assembly, core intrinsics are used instead of inline assembly if possible.
See the [`atomic128` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/atomic128/README.md) for details.
-## Optional features
-
-- **`fallback`** *(enabled by default)*<br>
- Enable fallback implementations.
-
- Disabling this allows only atomic types for which the platform natively supports atomic operations.
-
-- <a name="optional-features-float"></a>**`float`**<br>
- Provide `AtomicF{32,64}`.
-
- Note that most of `fetch_*` operations of atomic floats are implemented using CAS loops, which can be slower than equivalent operations of atomic integers. ([GPU targets have atomic instructions for float, so we plan to use these instructions for GPU targets in the future.](https://github.com/taiki-e/portable-atomic/issues/34))
-
-- **`std`**<br>
- Use `std`.
-
-- <a name="optional-features-require-cas"></a>**`require-cas`**<br>
- Emit compile error if atomic CAS is not available. See [Usage](#usage) section and [#100](https://github.com/taiki-e/portable-atomic/pull/100) for more.
-
-- <a name="optional-features-serde"></a>**`serde`**<br>
- Implement `serde::{Serialize,Deserialize}` for atomic types.
-
- Note:
- - The MSRV when this feature is enabled depends on the MSRV of [serde].
+## <a name="optional-features"></a><a name="optional-cfg"></a>Optional features/cfgs
-- <a name="optional-features-critical-section"></a>**`critical-section`**<br>
- When this feature is enabled, this crate uses [critical-section] to provide atomic CAS for targets where
- it is not natively available. When enabling it, you should provide a suitable critical section implementation
- for the current target, see the [critical-section] documentation for details on how to do so.
+portable-atomic provides features and cfgs to allow enabling specific APIs and customizing its behavior.
- `critical-section` support is useful to get atomic CAS when the [`unsafe-assume-single-core` feature](#optional-features-unsafe-assume-single-core) can't be used,
- such as multi-core targets, unprivileged code running under some RTOS, or environments where disabling interrupts
- needs extra care due to e.g. real-time requirements.
-
- Note that with the `critical-section` feature, critical sections are taken for all atomic operations, while with
- [`unsafe-assume-single-core` feature](#optional-features-unsafe-assume-single-core) some operations don't require disabling interrupts (loads and stores, but
- additionally on MSP430 `add`, `sub`, `and`, `or`, `xor`, `not`). Therefore, for better performance, if
- all the `critical-section` implementation for your target does is disable interrupts, prefer using
- `unsafe-assume-single-core` feature instead.
+Some options have both a feature and a cfg. When both exist, it indicates that the feature does not follow Cargo's recommendation that [features should be additive](https://doc.rust-lang.org/nightly/cargo/reference/features.html#feature-unification). Therefore, the maintainer's recommendation is to use cfg instead of feature. However, in the embedded ecosystem, it is very common to use features in such places, so these options provide both so you can choose based on your preference.
- Note:
- - The MSRV when this feature is enabled depends on the MSRV of [critical-section].
- - It is usually *not* recommended to always enable this feature in dependencies of the library.
+<details>
+<summary>How to enable cfg (click to show)</summary>
- Enabling this feature will prevent the end user from having the chance to take advantage of other (potentially) efficient implementations ([Implementations provided by `unsafe-assume-single-core` feature, default implementations on MSP430 and AVR](#optional-features-unsafe-assume-single-core), implementation proposed in [#60], etc. Other systems may also be supported in the future).
-
- The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature. (However, it may make sense to enable this feature by default for libraries specific to a platform where other implementations are known not to work.)
-
- As an example, the end-user's `Cargo.toml` that uses a crate that provides a critical-section implementation and a crate that depends on portable-atomic as an option would be expected to look like this:
-
- ```toml
- [dependencies]
- portable-atomic = { version = "1", default-features = false, features = ["critical-section"] }
- crate-provides-critical-section-impl = "..."
- crate-uses-portable-atomic-as-feature = { version = "...", features = ["portable-atomic"] }
- ```
-
-- <a name="optional-features-unsafe-assume-single-core"></a>**`unsafe-assume-single-core`**<br>
- Assume that the target is single-core.
- When this feature is enabled, this crate provides atomic CAS for targets where atomic CAS is not available in the standard library by disabling interrupts.
+One of the ways to enable cfg is to set [rustflags in the cargo config](https://doc.rust-lang.org/cargo/reference/config.html#targettriplerustflags):
- This feature is `unsafe`, and note the following safety requirements:
- - Enabling this feature for multi-core systems is always **unsound**.
- - This uses privileged instructions to disable interrupts, so it usually doesn't work on unprivileged mode.
- Enabling this feature in an environment where privileged instructions are not available, or if the instructions used are not sufficient to disable interrupts in the system, it is also usually considered **unsound**, although the details are system-dependent.
+```toml
+# .cargo/config.toml
+[target.<target>]
+rustflags = ["--cfg", "portable_atomic_unsafe_assume_single_core"]
+```
- The following are known cases:
- - On pre-v6 Arm, this disables only IRQs by default. For many systems (e.g., GBA) this is enough. If the system need to disable both IRQs and FIQs, you need to enable the `disable-fiq` feature together.
- - On RISC-V without A-extension, this generates code for machine-mode (M-mode) by default. If you enable the `s-mode` together, this generates code for supervisor-mode (S-mode). In particular, `qemu-system-riscv*` uses [OpenSBI](https://github.com/riscv-software-src/opensbi) as the default firmware.
+Or set environment variable:
- See also the [`interrupt` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/interrupt/README.md).
+```sh
+RUSTFLAGS="--cfg portable_atomic_unsafe_assume_single_core" cargo ...
+```
- Consider using the [`critical-section` feature](#optional-features-critical-section) for systems that cannot use this feature.
+</details>
- It is **very strongly discouraged** to enable this feature in libraries that depend on `portable-atomic`. The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature. (However, it may make sense to enable this feature by default for libraries specific to a platform where it is guaranteed to always be sound, for example in a hardware abstraction layer targeting a single-core chip.)
+- <a name="optional-features-fallback"></a>**`fallback` feature** *(enabled by default)*<br>
+ Enable fallback implementations.
- Armv6-M (thumbv6m), pre-v6 Arm (e.g., thumbv4t, thumbv5te), RISC-V without A-extension, and Xtensa are currently supported.
+ This enables atomic types with larger than the width supported by atomic instructions available on the current target. If the current target [supports 128-bit atomics](#128-bit-atomics-support), this is no-op.
- Since all MSP430 and AVR are single-core, we always provide atomic CAS for them without this feature.
+ This uses fallback implementation that using global locks by default. The following features/cfgs change this behavior:
+ - [`unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg](#optional-features-unsafe-assume-single-core): Use fallback implementations that disabling interrupts instead of using global locks.
+ - If your target is single-core and calling interrupt disable instructions is safe, this is a safer and more efficient option.
+ - [`unsafe-assume-privileged` feature / `portable_atomic_unsafe_assume_privileged` cfg](#optional-features-unsafe-assume-privileged): Use fallback implementations that using global locks with disabling interrupts.
+ - If your target is multi-core and calling interrupt disable instructions is safe, this is a safer option.
- Enabling this feature for targets that have atomic CAS will result in a compile error.
+- <a name="optional-features-float"></a>**`float` feature**<br>
+ Provide `AtomicF{32,64}`.
- Feel free to submit an issue if your target is not supported yet.
+ If you want atomic types for unstable float types ([`f16` and `f128`](https://github.com/rust-lang/rust/issues/116909)), enable unstable cfg (`portable_atomic_unstable_f16` cfg for `AtomicF16`, `portable_atomic_unstable_f128` cfg for `AtomicF128`, [there is no possibility that both feature and cfg will be provided for unstable options.](https://github.com/taiki-e/portable-atomic/pull/200#issuecomment-2682252991)).
-## Optional cfg
+> [!NOTE]
+> - Atomic float's `fetch_{add,sub,min,max}` are usually implemented using CAS loops, which can be slower than equivalent operations of atomic integers. As an exception, AArch64 with FEAT_LSFE and GPU targets have atomic float instructions and we use them on AArch64 when `lsfe` target feature is available at compile-time. We [plan to use atomic float instructions for GPU targets as well in the future.](https://github.com/taiki-e/portable-atomic/issues/34)
+> - Unstable cfgs are outside of the normal semver guarantees and minor or patch versions of portable-atomic may make breaking changes to them at any time.
-One of the ways to enable cfg is to set [rustflags in the cargo config](https://doc.rust-lang.org/cargo/reference/config.html#targettriplerustflags):
+- <a name="optional-features-std"></a>**`std` feature**<br>
+ Use `std`.
-```toml
-# .cargo/config.toml
-[target.<target>]
-rustflags = ["--cfg", "portable_atomic_no_outline_atomics"]
-```
+- <a name="optional-features-require-cas"></a>**`require-cas` feature**<br>
+ Emit compile error if atomic CAS is not available. See [Usage](#usage) section for usage of this feature.
-Or set environment variable:
+- <a name="optional-features-serde"></a>**`serde` feature**<br>
+ Implement `serde::{Serialize,Deserialize}` for atomic types.
-```sh
-RUSTFLAGS="--cfg portable_atomic_no_outline_atomics" cargo ...
-```
+ Note:
+ - The MSRV when this feature is enabled depends on the MSRV of [serde].
-- <a name="optional-cfg-unsafe-assume-single-core"></a>**`--cfg portable_atomic_unsafe_assume_single_core`**<br>
- Since 1.4.0, this cfg is an alias of [`unsafe-assume-single-core` feature](#optional-features-unsafe-assume-single-core).
+- <a name="optional-features-critical-section"></a>**`critical-section` feature**<br>
+ Use [critical-section] to provide atomic CAS for targets where atomic CAS is not available in the standard library.
- Originally, we were providing these as cfgs instead of features, but based on a strong request from the embedded ecosystem, we have agreed to provide them as features as well. See [#94](https://github.com/taiki-e/portable-atomic/pull/94) for more.
+ `critical-section` support is useful to get atomic CAS when the [`unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg)](#optional-features-unsafe-assume-single-core) can't be used,
+ such as multi-core targets, unprivileged code running under some RTOS, or environments where disabling interrupts
+ needs extra care due to e.g. real-time requirements.
-- <a name="optional-cfg-no-outline-atomics"></a>**`--cfg portable_atomic_no_outline_atomics`**<br>
+> [!NOTE]
+> - When enabling this feature, you should provide a suitable critical section implementation for the current target, see the [critical-section] documentation for details on how to do so.
+> - With this feature, critical sections are taken for all atomic operations, while with `unsafe-assume-single-core` feature [some operations](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/interrupt/README.md#no-disable-interrupts) don't require disabling interrupts. Therefore, for better performance, if all the `critical-section` implementation for your target does is disable interrupts, prefer using `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg) instead.
+> - It is usually **discouraged** to always enable this feature in libraries that depend on `portable-atomic`.
+>
+> Enabling this feature will prevent the end user from having the chance to take advantage of other (potentially) efficient implementations (implementations provided by `unsafe-assume-single-core` feature mentioned above, implementation proposed in [#60], etc.). Also, targets that are currently unsupported may be supported in the future.
+>
+> The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature. (However, it may make sense to enable this feature by default for libraries specific to a platform where other implementations are known not to work.)
+>
+> See also [](https://github.com/matklad/once_cell/issues/264#issuecomment-2352654806).
+>
+> As an example, the end-user's `Cargo.toml` that uses a crate that provides a critical-section implementation and a crate that depends on portable-atomic as an option would be expected to look like this:
+>
+> ```toml
+> [dependencies]
+> portable-atomic = { version = "1", default-features = false, features = ["critical-section"] }
+> crate-provides-critical-section-impl = "..."
+> crate-uses-portable-atomic-as-feature = { version = "...", features = ["portable-atomic"] }
+> ```
+>
+> - Enabling both this feature and `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg) will result in a compile error.
+> - Enabling both this feature and `unsafe-assume-privileged` feature (or `portable_atomic_unsafe_assume_privileged` cfg) will result in a compile error.
+> - The MSRV when this feature is enabled depends on the MSRV of [critical-section].
+
+- <a name="optional-features-unsafe-assume-single-core"></a><a name="optional-cfg-unsafe-assume-single-core"></a>**`unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg**<br>
+ Assume that the target is single-core and privileged instructions required to disable interrupts are available.
+
+ - When this feature/cfg is enabled, this crate provides atomic CAS for targets where atomic CAS is not available in the standard library by disabling interrupts.
+ - When both this feature/cfg and enabled-by-default `fallback` feature is enabled, this crate provides atomic types with larger than the width supported by native instructions by disabling interrupts.
+
+> [!WARNING]
+> This feature/cfg is `unsafe`, and note the following safety requirements:
+> - Enabling this feature/cfg for multi-core systems is always **unsound**.
+>
+> - This uses privileged instructions to disable interrupts, so it usually doesn't work on unprivileged mode.
+>
+> Enabling this feature/cfg in an environment where privileged instructions are not available, or if the instructions used are not sufficient to disable interrupts in the system, it is also usually considered **unsound**, although the details are system-dependent.
+>
+> The following are known cases:
+> - On Arm (except for M-Profile architectures), this disables only IRQs by default. For many systems (e.g., GBA) this is enough. If the system need to disable both IRQs and FIQs, you need to enable the `disable-fiq` feature (or `portable_atomic_disable_fiq` cfg) together.
+> - On RISC-V, this generates code for machine-mode (M-mode) by default. If you enable the `s-mode` feature (or `portable_atomic_s_mode` cfg) together, this generates code for supervisor-mode (S-mode). In particular, `qemu-system-riscv*` uses [OpenSBI](https://github.com/riscv-software-src/opensbi) as the default firmware.
+
+Consider using the [`unsafe-assume-privileged` feature (or `portable_atomic_unsafe_assume_privileged` cfg)](#optional-features-unsafe-assume-privileged) for multi-core systems with atomic CAS.
+
+Consider using the [`critical-section` feature](#optional-features-critical-section) for systems that cannot use this feature/cfg.
+
+See also the [`interrupt` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/interrupt/README.md).
+
+> [!NOTE]
+> - It is **very strongly discouraged** to enable this feature/cfg in libraries that depend on `portable-atomic`.
+>
+> The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature/cfg. (However, it may make sense to enable this feature/cfg by default for libraries specific to a platform where it is guaranteed to always be sound, for example in a hardware abstraction layer targeting a single-core chip.)
+> - Enabling this feature/cfg for unsupported architectures will result in a compile error.
+> - Arm, RISC-V, and Xtensa are currently supported. (Since all MSP430 and AVR are single-core, we always provide atomic CAS for them without this feature/cfg.)
+> - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target is not supported yet.
+> - Enabling this feature/cfg for targets where privileged instructions are obviously unavailable (e.g., Linux) will result in a compile error.
+> - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target supports privileged instructions but the build rejected.
+> - Enabling both this feature/cfg and `critical-section` feature will result in a compile error.
+> - When both this feature/cfg and `unsafe-assume-privileged` feature (or `portable_atomic_unsafe_assume_privileged` cfg) are enabled, this feature/cfg is preferred.
+
+- <a name="optional-features-unsafe-assume-privileged"></a><a name="optional-cfg-unsafe-assume-privileged"></a>**`unsafe-assume-privileged` feature / `portable_atomic_unsafe_assume_privileged` cfg**<br>
+ Similar to `unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg, but only assumes about availability of privileged instructions required to disable interrupts.
+
+ - When both this feature/cfg and enabled-by-default `fallback` feature is enabled, this crate provides atomic types with larger than the width supported by native instructions by using global locks with disabling interrupts.
+
+> [!WARNING]
+> This feature/cfg is `unsafe`, and except for being sound in multi-core systems, this has the same safety requirements as [`unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg](#optional-features-unsafe-assume-single-core).
+
+> [!NOTE]
+> - It is **very strongly discouraged** to enable this feature/cfg in libraries that depend on `portable-atomic`.
+>
+> The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature/cfg. (However, it may make sense to enable this feature/cfg by default for libraries specific to a platform where it is guaranteed to always be sound, for example in a hardware abstraction layer.)
+> - Enabling this feature/cfg for unsupported targets will result in a compile error.
+> - This requires atomic CAS (`cfg(target_has_atomic = "ptr")` or `cfg_no_atomic_cas!`).
+> - Arm, RISC-V, and Xtensa are currently supported.
+> - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target is not supported yet.
+> - Enabling this feature/cfg for targets where privileged instructions are obviously unavailable (e.g., Linux) will result in a compile error.
+> - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target supports privileged instructions but the build rejected.
+> - Enabling both this feature/cfg and `critical-section` feature will result in a compile error.
+> - When both this feature/cfg and `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg) are enabled, `unsafe-assume-single-core` is preferred.
+
+- <a name="optional-cfg-no-outline-atomics"></a>**`portable_atomic_no_outline_atomics` cfg**<br>
Disable dynamic dispatching by run-time CPU feature detection.
- If dynamic dispatching by run-time CPU feature detection is enabled, it allows maintaining support for older CPUs while using features that are not supported on older CPUs, such as CMPXCHG16B (x86_64) and FEAT_LSE/FEAT_LSE2 (AArch64).
-
- Note:
- - Dynamic detection is currently only supported in x86_64, AArch64, Arm, RISC-V (disabled by default), Arm64EC, and powerpc64, otherwise it works the same as when this cfg is set.
- - If the required target features are enabled at compile-time, the atomic operations are inlined.
- - This is compatible with no-std (as with all features except `std`).
- - On some targets, run-time detection is disabled by default mainly for incomplete build environments, and can be enabled by `--cfg portable_atomic_outline_atomics`. (When both cfg are enabled, `*_no_*` cfg is preferred.)
- - Some AArch64 targets enable LLVM's `outline-atomics` target feature by default, so if you set this cfg, you may want to disable that as well. (portable-atomic's outline-atomics does not depend on the compiler-rt symbols, so even if you need to disable LLVM's outline-atomics, you may not need to disable portable-atomic's outline-atomics.)
+ Dynamic dispatching by run-time CPU feature detection allows maintaining support for older CPUs while using features that are not supported on older CPUs, such as CMPXCHG16B (x86_64) and FEAT_LSE/FEAT_LSE2 (AArch64).
See also the [`atomic128` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/atomic128/README.md).
+> [!NOTE]
+> - If the required target features are enabled at compile-time, dynamic dispatching is automatically disabled and the atomic operations are inlined.
+> - This is compatible with no-std (as with all features except `std`).
+> - On some targets, run-time detection is disabled by default mainly for compatibility with incomplete build environments or support for it is experimental, and can be enabled by `portable_atomic_outline_atomics` cfg. (When both cfg are enabled, `*_no_*` cfg is preferred.)
+> - Some AArch64 targets enable LLVM's `outline-atomics` target feature by default, so if you set this cfg, you may want to disable that as well. (However, portable-atomic's outline-atomics does not depend on the compiler-rt symbols, so even if you need to disable LLVM's outline-atomics, you may not need to disable portable-atomic's outline-atomics.)
+> - Dynamic detection is currently only supported in x86_64, AArch64, Arm, RISC-V, Arm64EC, and powerpc64. Enabling this cfg for unsupported architectures will result in a compile error.
+
## Related Projects
- [atomic-maybe-uninit]: Atomic operations on potentially uninitialized integers.
@@ -190,7 +233,7 @@ RUSTFLAGS="--cfg portable_atomic_no_outline_atomics" cargo ...
[rust-lang/rust#100650]: https://github.com/rust-lang/rust/issues/100650
[serde]: https://github.com/serde-rs/serde
-<!-- tidy:crate-doc:end -->
+<!-- tidy:sync-markdown-to-rustdoc:end -->
## License
### external/vendor/portable-atomic/build.rs
@@ -6,19 +6,22 @@
#[path = "version.rs"]
mod version;
-use self::version::{rustc_version, Version};
+use self::version::{Version, rustc_version};
-use std::{env, str};
+#[path = "src/gen/build.rs"]
+mod generated;
-include!("no_atomic.rs");
+use std::{env, str};
fn main() {
println!("cargo:rerun-if-changed=build.rs");
- println!("cargo:rerun-if-changed=no_atomic.rs");
+ println!("cargo:rerun-if-changed=src/gen/build.rs");
println!("cargo:rerun-if-changed=version.rs");
#[cfg(feature = "unsafe-assume-single-core")]
println!("cargo:rustc-cfg=portable_atomic_unsafe_assume_single_core");
+ #[cfg(feature = "unsafe-assume-privileged")]
+ println!("cargo:rustc-cfg=portable_atomic_unsafe_assume_privileged");
#[cfg(feature = "s-mode")]
println!("cargo:rustc-cfg=portable_atomic_s_mode");
#[cfg(feature = "force-amo")]
@@ -47,18 +50,18 @@ fn main() {
if version.minor >= 80 {
println!(
- r#"cargo:rustc-check-cfg=cfg(target_feature,values("experimental-zacas","fast-serialization","load-store-on-cond","distinct-ops","miscellaneous-extensions-3"))"#
+ r#"cargo:rustc-check-cfg=cfg(target_feature,values("lsfe","fast-serialization","load-store-on-cond","distinct-ops","rmw"))"#
);
// Custom cfgs set by build script. Not public API.
// grep -F 'cargo:rustc-cfg=' build.rs | grep -Ev '^ *//' | sed -E 's/^.*cargo:rustc-cfg=//; s/(=\\)?".*$//' | LC_ALL=C sort -u | tr '\n' ',' | sed -E 's/,$/\n/'
println!(
- "cargo:rustc-check-cfg=cfg(portable_atomic_disable_fiq,portable_atomic_force_amo,portable_atomic_ll_sc_rmw,portable_atomic_new_atomic_intrinsics,portable_atomic_no_asm,portable_atomic_no_asm_maybe_uninit,portable_atomic_no_atomic_64,portable_atomic_no_atomic_cas,portable_atomic_no_atomic_load_store,portable_atomic_no_atomic_min_max,portable_atomic_no_cfg_target_has_atomic,portable_atomic_no_cmpxchg16b_intrinsic,portable_atomic_no_cmpxchg16b_target_feature,portable_atomic_no_const_mut_refs,portable_atomic_no_const_raw_ptr_deref,portable_atomic_no_const_transmute,portable_atomic_no_core_unwind_safe,portable_atomic_no_diagnostic_namespace,portable_atomic_no_offset_of,portable_atomic_no_stronger_failure_ordering,portable_atomic_no_track_caller,portable_atomic_no_unsafe_op_in_unsafe_fn,portable_atomic_pre_llvm_15,portable_atomic_pre_llvm_16,portable_atomic_pre_llvm_18,portable_atomic_s_mode,portable_atomic_sanitize_thread,portable_atomic_target_feature,portable_atomic_unsafe_assume_single_core,portable_atomic_unstable_asm,portable_atomic_unstable_asm_experimental_arch,portable_atomic_unstable_cfg_target_has_atomic,portable_atomic_unstable_isa_attribute)"
+ "cargo:rustc-check-cfg=cfg(portable_atomic_atomic_intrinsics,portable_atomic_disable_fiq,portable_atomic_force_amo,portable_atomic_ll_sc_rmw,portable_atomic_no_asm,portable_atomic_no_asm_maybe_uninit,portable_atomic_no_atomic_64,portable_atomic_no_atomic_cas,portable_atomic_no_atomic_load_store,portable_atomic_no_atomic_min_max,portable_atomic_no_cfg_target_has_atomic,portable_atomic_no_cmpxchg16b_intrinsic,portable_atomic_no_cmpxchg16b_target_feature,portable_atomic_no_const_mut_refs,portable_atomic_no_const_raw_ptr_deref,portable_atomic_no_const_transmute,portable_atomic_no_core_unwind_safe,portable_atomic_no_diagnostic_namespace,portable_atomic_no_strict_provenance,portable_atomic_no_strict_provenance_atomic_ptr,portable_atomic_no_stronger_failure_ordering,portable_atomic_no_track_caller,portable_atomic_no_unsafe_op_in_unsafe_fn,portable_atomic_pre_llvm_15,portable_atomic_pre_llvm_16,portable_atomic_pre_llvm_18,portable_atomic_pre_llvm_20,portable_atomic_s_mode,portable_atomic_sanitize_thread,portable_atomic_target_feature,portable_atomic_unsafe_assume_privileged,portable_atomic_unsafe_assume_single_core,portable_atomic_unstable_asm,portable_atomic_unstable_asm_experimental_arch,portable_atomic_unstable_cfg_target_has_atomic,portable_atomic_unstable_isa_attribute)"
);
// TODO: handle multi-line target_feature_fallback
// grep -F 'target_feature_fallback("' build.rs | grep -Ev '^ *//' | sed -E 's/^.*target_feature_fallback\(//; s/",.*$/"/' | LC_ALL=C sort -u | tr '\n' ',' | sed -E 's/,$/\n/'
println!(
- r#"cargo:rustc-check-cfg=cfg(portable_atomic_target_feature,values("cmpxchg16b","distinct-ops","experimental-zacas","fast-serialization","load-store-on-cond","lse","lse128","lse2","mclass","miscellaneous-extensions-3","quadword-atomics","rcpc3","v6","zaamo","zabha"))"#
+ r#"cargo:rustc-check-cfg=cfg(portable_atomic_target_feature,values("cmpxchg16b","distinct-ops","fast-serialization","load-store-on-cond","lse","lse128","lse2","lsfe","mclass","miscellaneous-extensions-3","quadword-atomics","rcpc3","rmw","v6","v7","zaamo","zabha","zacas"))"#
);
}
@@ -114,10 +117,6 @@ fn main() {
if !version.probe(74, 2023, 8, 23) {
println!("cargo:rustc-cfg=portable_atomic_no_asm_maybe_uninit");
}
- // For test
- if version.minor < 77 {
- println!("cargo:rustc-cfg=portable_atomic_no_offset_of");
- }
// #[diagnostic] stabilized in Rust 1.78 (nightly-2024-03-09): https://github.com/rust-lang/rust/pull/119888
if !version.probe(78, 2024, 3, 8) {
println!("cargo:rustc-cfg=portable_atomic_no_diagnostic_namespace");
@@ -126,6 +125,25 @@ fn main() {
if !version.probe(83, 2024, 9, 15) {
println!("cargo:rustc-cfg=portable_atomic_no_const_mut_refs");
}
+ // strict_provenance/exposed_provenance APIs stabilized in Rust 1.84 (nightly-2024-10-22): https://github.com/rust-lang/rust/pull/130350
+ if !version.probe(84, 2024, 10, 21) {
+ println!("cargo:rustc-cfg=portable_atomic_no_strict_provenance");
+ }
+ // strict_provenance_atomic_ptr stabilized in Rust 1.91 (nightly-2024-10-22): https://github.com/rust-lang/rust/pull/145467
+ if !version.probe(91, 2025, 8, 30) {
+ println!("cargo:rustc-cfg=portable_atomic_no_strict_provenance_atomic_ptr");
+ }
+
+ // For Miri and ThreadSanitizer. (aarch64, arm64ec, s390x, powerpc64)
+ // https://github.com/rust-lang/rust/pull/97423 merged in Rust 1.64 (nightly-2022-06-30).
+ // https://github.com/rust-lang/rust/pull/141507 merged in Rust 1.89 (nightly-2025-05-31).
+ if version.nightly
+ && version.probe(64, 2022, 6, 29)
+ && !version.probe(89, 2025, 5, 30)
+ && (target_arch != "powerpc64" || version.llvm >= 15)
+ {
+ println!("cargo:rustc-cfg=portable_atomic_atomic_intrinsics");
+ }
// asm! on AArch64, Arm, RISC-V, x86, and x86_64 stabilized in Rust 1.59 (nightly-2021-12-16): https://github.com/rust-lang/rust/pull/91728
let no_asm = !version.probe(59, 2021, 12, 15);
@@ -140,9 +158,12 @@ fn main() {
// x86 intel syntax requires LLVM 10 (since Rust 1.53, the minimum
// external LLVM version is 10+: https://github.com/rust-lang/rust/pull/83387).
// The part of this feature we use has not been changed since nightly-2020-06-21
- // until it was stabilized in nightly-2021-12-16, so it can be safely enabled in
- // nightly, which is older than nightly-2021-12-16.
+ // until it was stabilized, so it can safely be enabled in nightly for that period.
println!("cargo:rustc-cfg=portable_atomic_unstable_asm");
+ if (target_arch == "riscv32" || target_arch == "riscv64") && version.minor < 55 {
+ // Clobber-only registers used in riscv_linux.rs require Rust 1.55 (https://github.com/rust-lang/rust/pull/86416).
+ println!("cargo:rustc-cfg=portable_atomic_no_outline_atomics");
+ }
}
println!("cargo:rustc-cfg=portable_atomic_no_asm");
} else {
@@ -156,8 +177,23 @@ fn main() {
{
// https://github.com/rust-lang/rust/pull/111331 merged in Rust 1.71 (nightly-2023-05-09).
// The part of this feature we use has not been changed since nightly-2023-05-09
- // until it was stabilized in nightly-2024-11-11, so it can be safely enabled in
- // nightly, which is older than nightly-2024-11-11.
+ // until it was stabilized, so it can safely be enabled in nightly for that period.
+ println!("cargo:rustc-cfg=portable_atomic_unstable_asm_experimental_arch");
+ } else {
+ println!("cargo:rustc-cfg=portable_atomic_no_asm");
+ }
+ }
+ }
+ "powerpc64" => {
+ // asm! on PowerPC stabilized in Rust 1.95 (nightly-2026-01-28): https://github.com/rust-lang/rust/pull/147996
+ if !version.probe(95, 2026, 1, 27) {
+ if version.nightly
+ && version.probe(60, 2022, 2, 12)
+ && is_allowed_feature("asm_experimental_arch")
+ {
+ // https://github.com/rust-lang/rust/pull/93868 merged in Rust 1.60 (nightly-2022-02-13).
+ // The part of this feature we use has not been changed since nightly-2022-02-13
+ // until it was stabilized, so it can safely be enabled in nightly for that period.
println!("cargo:rustc-cfg=portable_atomic_unstable_asm_experimental_arch");
} else {
println!("cargo:rustc-cfg=portable_atomic_no_asm");
@@ -174,34 +210,36 @@ fn main() {
&& version.probe(40, 2019, 10, 13)
&& is_allowed_feature("cfg_target_has_atomic")
{
- // This feature has not been changed since the change in Rust 1.40 (nightly-2019-10-14)
- // until it was stabilized in nightly-2022-02-11, so it can be safely enabled in
- // nightly, which is older than nightly-2022-02-11.
+ // The part of this feature we use has not been changed since nightly-2019-10-14
+ // until it was stabilized, so it can safely be enabled in nightly for that period.
println!("cargo:rustc-cfg=portable_atomic_unstable_cfg_target_has_atomic");
} else {
println!("cargo:rustc-cfg=portable_atomic_no_cfg_target_has_atomic");
let target = &*convert_custom_linux_target(target);
- if NO_ATOMIC_CAS.contains(&target) {
+ if generated::NO_ATOMIC_CAS.contains(&target) {
println!("cargo:rustc-cfg=portable_atomic_no_atomic_cas");
}
- if NO_ATOMIC_64.contains(&target) {
+ if generated::NO_ATOMIC_64.contains(&target) {
println!("cargo:rustc-cfg=portable_atomic_no_atomic_64");
} else {
// Otherwise, assuming `"max-atomic-width" == 64` or `"max-atomic-width" == 128`.
}
}
}
// We don't need to use convert_custom_linux_target here because all linux targets have atomics.
- if NO_ATOMIC.contains(&target) {
+ if generated::NO_ATOMIC.contains(&target) {
println!("cargo:rustc-cfg=portable_atomic_no_atomic_load_store");
}
- if version.llvm < 18 {
- println!("cargo:rustc-cfg=portable_atomic_pre_llvm_18");
- if version.llvm < 16 {
- println!("cargo:rustc-cfg=portable_atomic_pre_llvm_16");
- if version.llvm < 15 {
- println!("cargo:rustc-cfg=portable_atomic_pre_llvm_15");
+ if version.llvm < 20 {
+ println!("cargo:rustc-cfg=portable_atomic_pre_llvm_20");
+ if version.llvm < 18 {
+ println!("cargo:rustc-cfg=portable_atomic_pre_llvm_18");
+ if version.llvm < 16 {
+ println!("cargo:rustc-cfg=portable_atomic_pre_llvm_16");
+ if version.llvm < 15 {
+ println!("cargo:rustc-cfg=portable_atomic_pre_llvm_15");
+ }
}
}
}
@@ -216,14 +254,6 @@ fn main() {
// false positives in our code.
println!("cargo:rustc-cfg=portable_atomic_sanitize_thread");
}
-
- // https://github.com/rust-lang/rust/pull/93868 merged in Rust 1.60 (nightly-2022-02-13).
- if !no_asm
- && (target_arch == "powerpc64" && version.probe(60, 2022, 2, 12))
- && is_allowed_feature("asm_experimental_arch")
- {
- println!("cargo:rustc-cfg=portable_atomic_unstable_asm_experimental_arch");
- }
}
match target_arch {
@@ -243,43 +273,41 @@ fn main() {
// x86_64 Apple targets always support CMPXCHG16B:
// https://github.com/rust-lang/rust/blob/1.68.0/compiler/rustc_target/src/spec/x86_64_apple_darwin.rs#L8
// https://github.com/rust-lang/rust/blob/1.68.0/compiler/rustc_target/src/spec/apple_base.rs#L69-L70
- // (Since Rust 1.78, Windows (except Windows 7) targets also enable CMPXCHG16B, but
- // this branch is only used on pre-1.69 that cmpxchg16b_target_feature is unstable.)
+ // (Windows (except Windows 7, since Rust 1.78) and Fuchsia (since Rust 1.87) targets
+ // also enable CMPXCHG16B, but this branch is only used on pre-1.69 that
+ // cmpxchg16b_target_feature is unstable.)
// Script to get builtin targets that support CMPXCHG16B by default:
- // $ (for target in $(rustc --print target-list | grep -E '^x86_64'); do rustc --print cfg --target "${target}" | grep -Fq '"cmpxchg16b"' && printf '%s\n' "${target}"; done)
+ // $ (for target in $(rustc -Z unstable-options --print all-target-specs-json | jq -r '. | to_entries[] | if .value.arch == "x86_64" then .key else empty end'); do rustc --print cfg --target "${target}" | grep -Fq '"cmpxchg16b"' && printf '%s\n' "${target}"; done)
let is_apple = env::var("CARGO_CFG_TARGET_VENDOR").unwrap_or_default() == "apple";
- let has_cmpxchg16b = is_apple;
+ let cmpxchg16b = is_apple;
// LLVM recognizes this also as cx16 target feature: https://godbolt.org/z/KM3jz616j
// However, it is unlikely that rustc will support that name, so we ignore it.
- target_feature_fallback("cmpxchg16b", has_cmpxchg16b);
+ target_feature_fallback("cmpxchg16b", cmpxchg16b);
}
}
"aarch64" | "arm64ec" => {
- // For Miri and ThreadSanitizer.
- // https://github.com/rust-lang/rust/pull/97423 merged in Rust 1.64 (nightly-2022-06-30).
- if version.nightly && version.probe(64, 2022, 6, 29) {
- println!("cargo:rustc-cfg=portable_atomic_new_atomic_intrinsics");
- }
-
// target_feature "lse2"/"lse128"/"rcpc3" is unstable and available on rustc side since nightly-2024-08-30: https://github.com/rust-lang/rust/pull/128192
if !version.probe(82, 2024, 8, 29) || needs_target_feature_fallback(&version, None) {
// FEAT_LSE2 doesn't imply FEAT_LSE. FEAT_LSE128 implies FEAT_LSE but not FEAT_LSE2.
// AArch64 macOS always supports FEAT_LSE and FEAT_LSE2 because M1 is Armv8.4 with all features of Armv8.5 except FEAT_BTI:
- // https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/AArch64/AArch64Processors.td#L1203
- // https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/AArch64/AArch64Processors.td#L865
+ // https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/AArch64/AArch64Processors.td#L1558
+ // https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/AArch64/AArch64Processors.td#L1180
// Script to get builtin targets that support FEAT_LSE/FEAT_LSE2 by default:
- // $ (for target in $(rustc --print target-list | grep -E '^aarch64|^arm64'); do rustc --print cfg --target "${target}" | grep -Fq '"lse"' && printf '%s\n' "${target}"; done)
- // $ (for target in $(rustc --print target-list | grep -E '^aarch64|^arm64'); do rustc --print cfg --target "${target}" | grep -Fq '"lse2"' && printf '%s\n' "${target}"; done)
+ // $ (for target in $(rustc -Z unstable-options --print all-target-specs-json | jq -r '. | to_entries[] | if .value.arch == "aarch64" or .value.arch == "arm64ec" then .key else empty end'); do rustc --print cfg --target "${target}" | grep -Fq '"lse"' && printf '%s\n' "${target}"; done)
+ // $ (for target in $(rustc -Z unstable-options --print all-target-specs-json | jq -r '. | to_entries[] | if .value.arch == "aarch64" or .value.arch == "arm64ec" then .key else empty end'); do rustc --print cfg --target "${target}" | grep -Fq '"lse2"' && printf '%s\n' "${target}"; done)
let is_macos = target_os == "macos";
- let mut has_lse = is_macos;
+ let mut lse = is_macos;
target_feature_fallback("lse2", is_macos);
- has_lse |= target_feature_fallback("lse128", false);
+ lse |= target_feature_fallback("lse128", false);
target_feature_fallback("rcpc3", false);
// aarch64_target_feature stabilized in Rust 1.61.
if needs_target_feature_fallback(&version, Some(61)) {
- target_feature_fallback("lse", has_lse);
+ target_feature_fallback("lse", lse);
}
}
+ // As of Rust 1.85, target_feature "lsfe" is not available on rustc side:
+ // https://github.com/rust-lang/rust/blob/1.85.0/compiler/rustc_target/src/target_features.rs
+ target_feature_fallback("lsfe", false);
// As of Apple M1/M1 Pro, on Apple hardware, CAS-loop-based RMW is much slower than
// LL/SC-loop-based RMW: https://github.com/taiki-e/portable-atomic/pull/89
@@ -312,10 +340,10 @@ fn main() {
"v7r" | "v8r" | "v9r" => {} // rclass
"v6m" | "v7em" | "v7m" | "v8m" => mclass = true,
// arm-linux-androideabi is v5te
- // https://github.com/rust-lang/rust/blob/1.80.0/compiler/rustc_target/src/spec/targets/arm_linux_androideabi.rs#L18
+ // https://github.com/rust-lang/rust/blob/1.84.0/compiler/rustc_target/src/spec/targets/arm_linux_androideabi.rs#L18
_ if target == "arm-linux-androideabi" => subarch = "v5te",
// armeb-unknown-linux-gnueabi is v8 & aclass
- // https://github.com/rust-lang/rust/blob/1.80.0/compiler/rustc_target/src/spec/targets/armeb_unknown_linux_gnueabi.rs#L18
+ // https://github.com/rust-lang/rust/blob/1.84.0/compiler/rustc_target/src/spec/targets/armeb_unknown_linux_gnueabi.rs#L18
_ if target == "armeb-unknown-linux-gnueabi" => subarch = "v8",
// Legacy Arm architectures (pre-v7 except v6m) don't have *class target feature.
"" => subarch = "v6",
@@ -332,105 +360,153 @@ fn main() {
);
}
}
- let v6 = known
- && (subarch.starts_with("v6")
- || subarch.starts_with("v7")
- || subarch.starts_with("v8")
- || subarch.starts_with("v9"));
+ let mut v6 = known && subarch.starts_with("v6");
+ let mut v7 = known && subarch.starts_with("v7");
+ let (v8, v8m) = if known && (subarch.starts_with("v8") || subarch.starts_with("v9"))
+ {
+ // Armv8-M is not considered as v8 by LLVM.
+ // https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/core_arch/src/arm_shared/mod.rs
+ if mclass {
+ // Armv8-M Mainline is a superset of Armv7-M.
+ // Armv8-M Baseline is a superset of Armv6-M.
+ // That said, LLVM handles thumbv8m.main without v8m like v6m, not v7m: https://godbolt.org/z/Ph96v9zae
+ // TODO: Armv9-M has not yet been released,
+ // so it is not clear how it will be handled here.
+ (false, true)
+ } else {
+ (true, false)
+ }
+ } else {
+ (false, false)
+ };
+ v7 |= v8;
+ v6 |= v8m;
+ v6 |= target_feature_fallback("v7", v7);
target_feature_fallback("v6", v6);
target_feature_fallback("mclass", mclass);
}
}
"riscv32" | "riscv64" => {
- // zabha and zacas imply zaamo in GCC and Rust, but do not in LLVM (but enabling them
- // without zaamo or a is not allowed, so we can assume zaamo is available when zabha is enabled).
- // https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/TargetParser/RISCVISAInfo.cpp#L772-L778
- // https://github.com/gcc-mirror/gcc/blob/08693e29ec186fd7941d0b73d4d466388971fe2f/gcc/config/riscv/arch-canonicalize#L45-L46
+ // zabha and zacas imply zaamo in GCC, LLVM 20+, and Rust, but do not in LLVM 19.
+ // However, enabling them without zaamo or a is not allowed in LLVM 19, so we can assume
+ // zaamo is available when zabha is enabled).
+ // https://github.com/llvm/llvm-project/commit/956361ca080a689a96b6552d28681aaf0ad2f494
+ // https://github.com/gcc-mirror/gcc/commit/7b2b2e3d660edc8ef3a8cfbdfc2b0fd499459601
+ // https://github.com/gcc-mirror/gcc/commit/11c2453a16b725b7fb67778e1ab4636a51a1217d
// https://github.com/rust-lang/rust/pull/130877
- let mut has_zaamo = false;
- // As of rustc 1.80, target_feature "zacas" is not available on rustc side:
- // https://github.com/rust-lang/rust/blob/1.80.0/compiler/rustc_target/src/target_features.rs#L273
- if version.llvm == 19 {
+ let mut zaamo = false;
+ // target_feature "zacas" is available as unstable on rustc side
+ // since nightly-2025-02-26 (https://github.com/rust-lang/rust/pull/137417),
+ // and stabilized in Rust 1.94 (https://github.com/rust-lang/rust/pull/145948).
+ if !version.probe(87, 2025, 2, 25) || needs_target_feature_fallback(&version, Some(94))
+ {
// amocas.{w,d,q} (and amocas.{b,h} if zabha is also available)
// available as experimental since LLVM 17 https://github.com/llvm/llvm-project/commit/29f630a1ddcbb03caa31b5002f0cbc105ff3a869
- // attempted to make non-experimental in LLVM 19 https://github.com/llvm/llvm-project/commit/95aab69c109adf29e183090c25dc95c773215746
- // but reverted in https://github.com/llvm/llvm-project/commit/70e7d26e560173c8b9db4c75ab4a3004cd5f021a
- // check == 19 instead of range 17..=19 because it is more experimental in LLVM 17/18.
- // check == 19 instead of >= 19 because "experimental-zacas" feature
- // may no longer exist when it is marked as non-experimental in LLVM 20.
- // https://github.com/llvm/llvm-project/commit/614aeda93b2225c6eb42b00ba189ba7ca2585c60
- has_zaamo |= target_feature_fallback("experimental-zacas", false);
+ // available non-experimental since LLVM 20 https://github.com/llvm/llvm-project/commit/614aeda93b2225c6eb42b00ba189ba7ca2585c60
+ zaamo |= target_feature_fallback("zacas", false);
}
- // target_feature "zaamo"/"zabha" is unstable and available on rustc side since nightly-2024-10-02: https://github.com/rust-lang/rust/pull/130877
- if !version.probe(83, 2024, 10, 1) || needs_target_feature_fallback(&version, None) {
+ // target_feature "zaamo"/"zabha" is available as unstable on rustc side
+ // since nightly-2024-10-02 (https://github.com/rust-lang/rust/pull/130877),
+ // and stabilized in Rust 1.94 (https://github.com/rust-lang/rust/pull/145948).
+ if !version.probe(83, 2024, 10, 1) || needs_target_feature_fallback(&version, Some(94))
+ {
if version.llvm >= 19 {
// amo*.{b,h}
// available since LLVM 19 https://github.com/llvm/llvm-project/commit/89f87c387627150d342722b79c78cea2311cddf7 / https://github.com/llvm/llvm-project/commit/6b7444964a8d028989beee554a1f5c61d16a1cac
- has_zaamo |= target_feature_fallback("zabha", false);
+ zaamo |= target_feature_fallback("zabha", false);
}
// amo*.{w,d}
- target_feature_fallback("zaamo", has_zaamo);
+ target_feature_fallback("zaamo", zaamo);
}
}
"powerpc64" => {
// target_feature "quadword-atomics" is unstable and available on rustc side since nightly-2024-09-28: https://github.com/rust-lang/rust/pull/130873
if !version.probe(83, 2024, 9, 27) || needs_target_feature_fallback(&version, None) {
- let target_endian =
- env::var("CARGO_CFG_TARGET_ENDIAN").expect("CARGO_CFG_TARGET_ENDIAN not set");
- // powerpc64le is pwr8 by default https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/PowerPC/PPC.td#L702
- // See also https://github.com/rust-lang/rust/issues/59932
- let mut pwr8_features = target_endian == "little";
- if let Some(cpu) = &target_cpu() {
- if let Some(mut cpu_version) = strip_prefix(cpu, "pwr") {
+ let mut pwr8_features = false;
+ if let Some(cpu) = target_cpu() {
+ if let Some(mut cpu_version) = strip_prefix(&cpu, "pwr") {
cpu_version = strip_suffix(cpu_version, "x").unwrap_or(cpu_version); // for pwr5x and pwr6x
if let Ok(cpu_version) = cpu_version.parse::<u32>() {
pwr8_features = cpu_version >= 8;
}
} else {
- // https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/PowerPC/PPC.td#L702
- // https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/PowerPC/PPC.td#L483
+ // https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/PowerPC/PPC.td#L789
+ // https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/PowerPC/PPC.td#L557
// On the minimum external LLVM version of the oldest rustc version which we can use asm_experimental_arch
// on this target (see CI config for more), "future" is based on pwr10 features.
// https://github.com/llvm/llvm-project/blob/llvmorg-12.0.0/llvm/lib/Target/PowerPC/PPC.td#L370
pwr8_features = cpu == "future" || cpu == "ppc64le";
}
+ } else {
+ // powerpc64le is pwr8 by default https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/PowerPC/PPC.td#L789
+ // See also https://github.com/rust-lang/rust/issues/59932
+ pwr8_features = env::var("CARGO_CFG_TARGET_ENDIAN")
+ .expect("CARGO_CFG_TARGET_ENDIAN not set")
+ == "little";
}
- // power8 features: https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/PowerPC/PPC.td#L409
+ // power8 features: https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/PowerPC/PPC.td#L484
// lqarx and stqcx.
target_feature_fallback("quadword-atomics", pwr8_features);
}
}
"s390x" => {
- // https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/SystemZ/SystemZFeatures.td
let mut arch9_features = false; // z196+
let mut arch13_features = false; // z15+
if let Some(cpu) = target_cpu() {
// LLVM and GCC recognize the same names:
- // https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/SystemZ/SystemZProcessors.td
- // https://github.com/gcc-mirror/gcc/blob/releases/gcc-14.2.0/gcc/config/s390/s390.opt#L58-L125
- match &*cpu {
- "arch9" | "z196" | "arch10" | "zEC12" | "arch11" | "z13" | "arch12" | "z14" => {
- arch9_features = true;
+ // https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/SystemZ/SystemZProcessors.td
+ // https://github.com/gcc-mirror/gcc/blob/releases/gcc-15.2.0/gcc/config/s390/s390.opt#L58-L128
+ if let Some(arch_version) = strip_prefix(&cpu, "arch") {
+ if let Ok(arch_version) = arch_version.parse::<u32>() {
+ arch9_features = arch_version >= 9;
+ arch13_features = arch_version >= 13;
}
- "arch13" | "z15" | "arch14" | "z16" => {
- arch9_features = true;
- arch13_features = true;
+ } else {
+ match &*cpu {
+ "z196" | "zEC12" | "z13" | "z14" => arch9_features = true,
+ "z15" | "z16" | "z17" => {
+ arch9_features = true;
+ arch13_features = true;
+ }
+ _ => {}
}
- _ => {}
}
}
- // As of rustc 1.80, target_feature "fast-serialization"/"load-store-on-cond"/"distinct-ops"/"miscellaneous-extensions-3" is not available on rustc side:
- // https://github.com/rust-lang/rust/blob/1.80.0/compiler/rustc_target/src/target_features.rs
- // arch9 features: https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/SystemZ/SystemZFeatures.td#L103
+ // target_feature "miscellaneous-extensions-3" is available as unstable on rustc side
+ // since nightly-2025-06-05 (https://github.com/rust-lang/rust/pull/141250),
+ // and stabilized in Rust 1.93 (https://github.com/rust-lang/rust/pull/145656).
+ if !version.probe(89, 2025, 6, 4) || needs_target_feature_fallback(&version, Some(93)) {
+ // arch13 features: https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/SystemZ/SystemZFeatures.td#L303
+ // nand (nnr{,g}k), select (sel{,g}r), etc.
+ target_feature_fallback("miscellaneous-extensions-3", arch13_features);
+ }
+ // As of Rust 1.84, target_feature "fast-serialization"/"load-store-on-cond"/"distinct-ops"/"miscellaneous-extensions-3" is not available on rustc side:
+ // https://github.com/rust-lang/rust/blob/1.84.0/compiler/rustc_target/src/target_features.rs#L547
+ // arch9 features: https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/SystemZ/SystemZFeatures.td#L103
// bcr 14,0
target_feature_fallback("fast-serialization", arch9_features);
// {l,st}oc{,g}{,r}
target_feature_fallback("load-store-on-cond", arch9_features);
// {al,sl,n,o,x}{,g}rk
target_feature_fallback("distinct-ops", arch9_features);
- // arch13 features: https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/SystemZ/SystemZFeatures.td#L301
- // nand (nnr{,g}k), select (sel{,g}r), etc.
- target_feature_fallback("miscellaneous-extensions-3", arch13_features);
+ }
+ "avr" => {
+ // target_feature "rmw" will be added in https://github.com/rust-lang/rust/pull/146900
+ // https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/AVR/AVRDevices.td
+ let mut xmegau = false; // FamilyXMEGAU
+ if let Some(cpu) = target_cpu() {
+ match &*cpu {
+ "atxmega16a4u" | "atxmega16c4" | "atxmega32a4u" | "atxmega32c3"
+ | "atxmega32c4" | "atxmega32e5" | "atxmega16e5" | "atxmega8e5"
+ | "atxmega64a3u" | "atxmega64a4u" | "atxmega64b1" | "atxmega64b3"
+ | "atxmega64c3" | "atxmega64a1u" | "atxmega128a3u" | "atxmega128b1"
+ | "atxmega128b3" | "atxmega128c3" | "atxmega192a3u" | "atxmega192c3"
+ | "atxmega256a3u" | "atxmega256a3bu" | "atxmega256c3" | "atxmega384c3"
+ | "atxmega128a1u" | "atxmega128a4u" => xmegau = true,
+ _ => {}
+ }
+ }
+ target_feature_fallback("rmw", xmegau);
}
_ => {}
}
@@ -510,7 +586,7 @@ fn is_allowed_feature(name: &str) -> bool {
allowed
}
-// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.14/build-common.rs.
+// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/build-common.rs.
//
// The target triplets have the form of 'arch-vendor-system'.
//
@@ -529,18 +605,10 @@ fn convert_custom_linux_target(target: &str) -> String {
// str::strip_prefix requires Rust 1.45
#[must_use]
fn strip_prefix<'a>(s: &'a str, pat: &str) -> Option<&'a str> {
- if s.starts_with(pat) {
- Some(&s[pat.len()..])
- } else {
- None
- }
+ if s.starts_with(pat) { Some(&s[pat.len()..]) } else { None }
}
// str::strip_suffix requires Rust 1.45
#[must_use]
fn strip_suffix<'a>(s: &'a str, pat: &str) -> Option<&'a str> {
- if s.ends_with(pat) {
- Some(&s[..s.len() - pat.len()])
- } else {
- None
- }
+ if s.ends_with(pat) { Some(&s[..s.len() - pat.len()]) } else { None }
}
### external/vendor/portable-atomic/matrix-old.json
@@ -0,0 +1,398 @@
+{
+ "include": [
+ {
+ "rust": "1.59",
+ "target": "x86_64-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly-2021-08-21",
+ "target": "x86_64-unknown-linux-gnu"
+ },
+ {
+ "rust": "1.56",
+ "target": "x86_64-unknown-linux-gnu"
+ },
+ {
+ "rust": "stable",
+ "target": "x86_64-unknown-linux-gnu"
+ },
+ {
+ "rust": "beta",
+ "target": "x86_64-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "x86_64-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "x86_64-unknown-linux-gnu",
+ "flags": "-C panic=abort -Z panic_abort_tests"
+ },
+ {
+ "rust": "nightly",
+ "target": "x86_64-unknown-linux-gnu",
+ "flags": "-Z codegen-backend=cranelift"
+ },
+ {
+ "rust": "nightly",
+ "target": "x86_64-unknown-linux-gnu",
+ "flags": "-Z codegen-backend=gcc"
+ },
+ {
+ "rust": "nightly",
+ "target": "x86_64-apple-darwin",
+ "os": "macos-15-intel"
+ },
+ {
+ "rust": "nightly",
+ "target": "x86_64-apple-darwin",
+ "os": "macos-latest"
+ },
+ {
+ "rust": "1.59",
+ "target": "x86_64-pc-windows-msvc",
+ "os": "windows-latest"
+ },
+ {
+ "rust": "nightly-2021-08-21",
+ "target": "x86_64-pc-windows-msvc",
+ "os": "windows-latest"
+ },
+ {
+ "rust": "1.56",
+ "target": "x86_64-pc-windows-msvc",
+ "os": "windows-latest"
+ },
+ {
+ "rust": "stable",
+ "target": "x86_64-pc-windows-msvc",
+ "os": "windows-latest"
+ },
+ {
+ "rust": "beta",
+ "target": "x86_64-pc-windows-msvc",
+ "os": "windows-latest"
+ },
+ {
+ "rust": "nightly",
+ "target": "x86_64-pc-windows-msvc",
+ "os": "windows-latest"
+ },
+ {
+ "rust": "nightly-x86_64-pc-windows-gnu",
+ "target": "x86_64-pc-windows-gnu",
+ "os": "windows-latest"
+ },
+ {
+ "rust": "nightly",
+ "target": "i586-unknown-linux-gnu"
+ },
+ {
+ "rust": "1.59",
+ "target": "i686-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly-2021-08-21",
+ "target": "i686-unknown-linux-gnu"
+ },
+ {
+ "rust": "stable",
+ "target": "i686-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "i686-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly-i686-pc-windows-msvc",
+ "target": "i686-pc-windows-msvc",
+ "os": "windows-latest"
+ },
+ {
+ "rust": "nightly-i686-pc-windows-gnu",
+ "target": "i686-pc-windows-gnu",
+ "os": "windows-latest"
+ },
+ {
+ "rust": "1.59",
+ "target": "aarch64-unknown-linux-gnu",
+ "os": "ubuntu-24.04-arm"
+ },
+ {
+ "rust": "nightly-2021-08-21",
+ "target": "aarch64-unknown-linux-gnu",
+ "os": "ubuntu-24.04-arm"
+ },
+ {
+ "rust": "stable",
+ "target": "aarch64-unknown-linux-gnu",
+ "os": "ubuntu-24.04-arm"
+ },
+ {
+ "rust": "beta",
+ "target": "aarch64-unknown-linux-gnu",
+ "os": "ubuntu-24.04-arm"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-unknown-linux-gnu",
+ "os": "ubuntu-24.04-arm"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-unknown-linux-gnu",
+ "os": "ubuntu-24.04-arm",
+ "flags": "-Z codegen-backend=cranelift"
+ },
+ {
+ "rust": "1.59",
+ "target": "aarch64-apple-darwin",
+ "os": "macos-latest"
+ },
+ {
+ "rust": "nightly-2021-08-21",
+ "target": "aarch64-apple-darwin",
+ "os": "macos-latest"
+ },
+ {
+ "rust": "stable",
+ "target": "aarch64-apple-darwin",
+ "os": "macos-latest"
+ },
+ {
+ "rust": "beta",
+ "target": "aarch64-apple-darwin",
+ "os": "macos-latest"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-apple-darwin",
+ "os": "macos-latest"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-apple-ios-macabi",
+ "os": "macos-latest"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-pc-windows-msvc",
+ "os": "windows-11-arm"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-pc-windows-gnullvm",
+ "os": "windows-11-arm"
+ },
+ {
+ "rust": "stable",
+ "target": "aarch64-unknown-linux-musl"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-unknown-linux-musl"
+ },
+ {
+ "rust": "stable",
+ "target": "aarch64-unknown-linux-musl",
+ "flags": "-C target-feature=-crt-static"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-unknown-linux-musl",
+ "flags": "-C target-feature=-crt-static"
+ },
+ {
+ "rust": "stable",
+ "target": "aarch64-linux-android"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64-linux-android"
+ },
+ {
+ "rust": "nightly-2024-07-31",
+ "target": "aarch64_be-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "aarch64_be-unknown-linux-gnu"
+ },
+ {
+ "rust": "1.84",
+ "target": "arm64ec-pc-windows-msvc",
+ "os": "windows-11-arm"
+ },
+ {
+ "rust": "stable",
+ "target": "arm64ec-pc-windows-msvc",
+ "os": "windows-11-arm"
+ },
+ {
+ "rust": "nightly",
+ "target": "arm64ec-pc-windows-msvc",
+ "os": "windows-11-arm"
+ },
+ {
+ "rust": "1.59",
+ "target": "armv5te-unknown-linux-gnueabi"
+ },
+ {
+ "rust": "nightly-2021-08-21",
+ "target": "armv5te-unknown-linux-gnueabi"
+ },
+ {
+ "rust": "stable",
+ "target": "armv5te-unknown-linux-gnueabi"
+ },
+ {
+ "rust": "nightly",
+ "target": "armv5te-unknown-linux-gnueabi"
+ },
+ {
+ "rust": "nightly",
+ "target": "arm-unknown-linux-gnueabi"
+ },
+ {
+ "rust": "nightly",
+ "target": "armv7-unknown-linux-gnueabi"
+ },
+ {
+ "rust": "nightly",
+ "target": "armv7-unknown-linux-gnueabihf",
+ "os": "ubuntu-24.04-arm"
+ },
+ {
+ "rust": "nightly",
+ "target": "armv7-unknown-linux-gnueabihf"
+ },
+ {
+ "rust": "nightly",
+ "target": "thumbv7neon-unknown-linux-gnueabihf",
+ "os": "ubuntu-24.04-arm"
+ },
+ {
+ "rust": "nightly",
+ "target": "thumbv7neon-unknown-linux-gnueabihf"
+ },
+ {
+ "rust": "nightly",
+ "target": "armeb-unknown-linux-gnueabi",
+ "os": "ubuntu-22.04"
+ },
+ {
+ "rust": "nightly-2024-08-30",
+ "target": "arm-linux-androideabi"
+ },
+ {
+ "rust": "nightly",
+ "target": "loongarch64-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "mips-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "mipsel-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "mips64-unknown-linux-gnuabi64"
+ },
+ {
+ "rust": "nightly",
+ "target": "mips64el-unknown-linux-gnuabi64"
+ },
+ {
+ "rust": "nightly",
+ "target": "mipsisa32r6-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "mipsisa32r6el-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "mipsisa64r6-unknown-linux-gnuabi64"
+ },
+ {
+ "rust": "nightly",
+ "target": "mipsisa64r6el-unknown-linux-gnuabi64"
+ },
+ {
+ "rust": "nightly",
+ "target": "powerpc-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly-2022-02-13",
+ "target": "powerpc64-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "powerpc64-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly-2022-02-13",
+ "target": "powerpc64le-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "powerpc64le-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly-2022-08-12",
+ "target": "riscv32gc-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "riscv32gc-unknown-linux-gnu"
+ },
+ {
+ "rust": "1.59",
+ "target": "riscv64gc-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly-2021-08-21",
+ "target": "riscv64gc-unknown-linux-gnu"
+ },
+ {
+ "rust": "stable",
+ "target": "riscv64gc-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "riscv64gc-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "riscv64gc-unknown-linux-gnu",
+ "flags": "-Z codegen-backend=cranelift"
+ },
+ {
+ "rust": "1.84",
+ "target": "s390x-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly-2023-05-09",
+ "target": "s390x-unknown-linux-gnu"
+ },
+ {
+ "rust": "stable",
+ "target": "s390x-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "s390x-unknown-linux-gnu"
+ },
+ {
+ "rust": "nightly",
+ "target": "sparc64-unknown-linux-gnu"
+ }
+ ]
+}
### external/vendor/portable-atomic/src/cfgs.rs
@@ -10,6 +10,7 @@
target_arch = "riscv32",
target_arch = "riscv64",
feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
)),
)))]
#[macro_use]
@@ -43,6 +44,7 @@ mod atomic_8_16_macros {
target_arch = "riscv32",
target_arch = "riscv64",
feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
)),
))]
#[macro_use]
@@ -79,6 +81,7 @@ mod atomic_8_16_macros {
target_arch = "riscv32",
target_arch = "riscv64",
feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
)),
)),
))]
@@ -105,6 +108,7 @@ mod atomic_32_macros {
target_arch = "riscv32",
target_arch = "riscv64",
feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
)),
)),
)))]
@@ -129,14 +133,20 @@ mod atomic_32_macros {
feature = "fallback",
any(
not(portable_atomic_no_atomic_cas),
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
),
),
not(portable_atomic_no_atomic_64),
not(any(target_pointer_width = "16", target_pointer_width = "32")),
+ all(
+ target_arch = "riscv32",
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(target_feature = "zacas", portable_atomic_target_feature = "zacas"),
+ ),
))
)]
#[cfg_attr(
@@ -146,28 +156,19 @@ mod atomic_32_macros {
feature = "fallback",
any(
target_has_atomic = "ptr",
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
),
),
target_has_atomic = "64",
not(any(target_pointer_width = "16", target_pointer_width = "32")),
all(
target_arch = "riscv32",
not(any(miri, portable_atomic_sanitize_thread)),
- not(portable_atomic_no_asm),
- any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- any(test, portable_atomic_outline_atomics), // TODO(riscv): currently disabled by default
- any(target_os = "linux", target_os = "android"),
- ),
- ),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(target_feature = "zacas", portable_atomic_target_feature = "zacas"),
),
))
)]
@@ -191,14 +192,20 @@ mod atomic_64_macros {
feature = "fallback",
any(
not(portable_atomic_no_atomic_cas),
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
),
),
not(portable_atomic_no_atomic_64),
not(any(target_pointer_width = "16", target_pointer_width = "32")),
+ all(
+ target_arch = "riscv32",
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(target_feature = "zacas", portable_atomic_target_feature = "zacas"),
+ ),
)))
)]
#[cfg_attr(
@@ -208,28 +215,19 @@ mod atomic_64_macros {
feature = "fallback",
any(
target_has_atomic = "ptr",
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
),
),
target_has_atomic = "64",
not(any(target_pointer_width = "16", target_pointer_width = "32")),
all(
target_arch = "riscv32",
not(any(miri, portable_atomic_sanitize_thread)),
- not(portable_atomic_no_asm),
- any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- any(test, portable_atomic_outline_atomics), // TODO(riscv): currently disabled by default
- any(target_os = "linux", target_os = "android"),
- ),
- ),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(target_feature = "zacas", portable_atomic_target_feature = "zacas"),
),
)))
)]
@@ -252,94 +250,75 @@ mod atomic_64_macros {
cfg(any(
all(
target_arch = "aarch64",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
),
- all(target_arch = "arm64ec", not(portable_atomic_no_asm)),
+ all(
+ target_arch = "arm64ec",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
+ not(portable_atomic_no_asm),
+ ),
all(
target_arch = "x86_64",
not(all(
any(miri, portable_atomic_sanitize_thread),
portable_atomic_no_cmpxchg16b_intrinsic,
)),
any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
- any(
- target_feature = "cmpxchg16b",
- portable_atomic_target_feature = "cmpxchg16b",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- not(any(target_env = "sgx", miri)),
- ),
- ),
+ any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
),
all(
target_arch = "riscv64",
- not(portable_atomic_no_asm),
- any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- any(test, portable_atomic_outline_atomics), // TODO(riscv): currently disabled by default
- any(target_os = "linux", target_os = "android"),
- not(any(miri, portable_atomic_sanitize_thread)),
- ),
- ),
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(target_feature = "zacas", portable_atomic_target_feature = "zacas"),
),
all(
target_arch = "powerpc64",
- portable_atomic_unstable_asm_experimental_arch,
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
+ not(portable_atomic_no_asm),
any(
target_feature = "quadword-atomics",
portable_atomic_target_feature = "quadword-atomics",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- any(
- all(
- target_os = "linux",
- any(
- all(
- target_env = "gnu",
- any(target_endian = "little", not(target_feature = "crt-static")),
- ),
- all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
- ),
- portable_atomic_outline_atomics,
- ),
- ),
- target_os = "android",
- target_os = "freebsd",
- target_os = "openbsd",
- ),
- not(any(miri, portable_atomic_sanitize_thread)),
- ),
),
),
- all(target_arch = "s390x", not(portable_atomic_no_asm)),
+ all(
+ target_arch = "s390x",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
+ not(portable_atomic_no_asm),
+ ),
))
)]
#[cfg_attr(
all(feature = "fallback", portable_atomic_no_cfg_target_has_atomic),
cfg(any(
not(portable_atomic_no_atomic_cas),
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
))
)]
#[cfg_attr(
all(feature = "fallback", not(portable_atomic_no_cfg_target_has_atomic)),
cfg(any(
target_has_atomic = "ptr",
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
))
)]
#[macro_use]
@@ -360,94 +339,75 @@ mod atomic_128_macros {
cfg(not(any(
all(
target_arch = "aarch64",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
),
- all(target_arch = "arm64ec", not(portable_atomic_no_asm)),
+ all(
+ target_arch = "arm64ec",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
+ not(portable_atomic_no_asm),
+ ),
all(
target_arch = "x86_64",
not(all(
any(miri, portable_atomic_sanitize_thread),
portable_atomic_no_cmpxchg16b_intrinsic,
)),
any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
- any(
- target_feature = "cmpxchg16b",
- portable_atomic_target_feature = "cmpxchg16b",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- not(any(target_env = "sgx", miri)),
- ),
- ),
+ any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
),
all(
target_arch = "riscv64",
- not(portable_atomic_no_asm),
- any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- any(test, portable_atomic_outline_atomics), // TODO(riscv): currently disabled by default
- any(target_os = "linux", target_os = "android"),
- not(any(miri, portable_atomic_sanitize_thread)),
- ),
- ),
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(target_feature = "zacas", portable_atomic_target_feature = "zacas"),
),
all(
target_arch = "powerpc64",
- portable_atomic_unstable_asm_experimental_arch,
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
+ not(portable_atomic_no_asm),
any(
target_feature = "quadword-atomics",
portable_atomic_target_feature = "quadword-atomics",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- any(
- all(
- target_os = "linux",
- any(
- all(
- target_env = "gnu",
- any(target_endian = "little", not(target_feature = "crt-static")),
- ),
- all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
- ),
- portable_atomic_outline_atomics,
- ),
- ),
- target_os = "android",
- target_os = "freebsd",
- target_os = "openbsd",
- ),
- not(any(miri, portable_atomic_sanitize_thread)),
- ),
),
),
- all(target_arch = "s390x", not(portable_atomic_no_asm)),
+ all(
+ target_arch = "s390x",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
+ not(portable_atomic_no_asm),
+ ),
)))
)]
#[cfg_attr(
all(feature = "fallback", portable_atomic_no_cfg_target_has_atomic),
cfg(not(any(
not(portable_atomic_no_atomic_cas),
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
)))
)]
#[cfg_attr(
all(feature = "fallback", not(portable_atomic_no_cfg_target_has_atomic)),
cfg(not(any(
target_has_atomic = "ptr",
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
)))
)]
#[macro_use]
@@ -468,20 +428,20 @@ mod atomic_128_macros {
portable_atomic_no_cfg_target_has_atomic,
cfg(any(
not(portable_atomic_no_atomic_cas),
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
))
)]
#[cfg_attr(
not(portable_atomic_no_cfg_target_has_atomic),
cfg(any(
target_has_atomic = "ptr",
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
))
)]
#[macro_use]
@@ -512,20 +472,20 @@ mod atomic_cas_macros {
portable_atomic_no_cfg_target_has_atomic,
cfg(not(any(
not(portable_atomic_no_atomic_cas),
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
)))
)]
#[cfg_attr(
not(portable_atomic_no_cfg_target_has_atomic),
cfg(not(any(
target_has_atomic = "ptr",
- portable_atomic_unsafe_assume_single_core,
- feature = "critical-section",
target_arch = "avr",
target_arch = "msp430",
+ feature = "critical-section",
+ portable_atomic_unsafe_assume_single_core,
)))
)]
#[macro_use]
@@ -627,7 +587,7 @@ mod check {
crate::cfg_no_atomic_cas! { type __AtomicPtr = (); }
#[allow(unused_imports)]
use self::{
- _Atomic128 as _, _Atomic16 as _, _Atomic32 as _, _Atomic64 as _, _Atomic8 as _,
- _AtomicPtr as _, __AtomicPtr as _,
+ __AtomicPtr as _, _Atomic8 as _, _Atomic16 as _, _Atomic32 as _, _Atomic64 as _,
+ _Atomic128 as _, _AtomicPtr as _,
};
}
### external/vendor/portable-atomic/src/gen/build.rs
@@ -1,11 +1,11 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
-// This file is @generated by no_atomic.sh.
+// This file is @generated by target_spec.sh.
// It is not intended for manual editing.
// Note: This is the list as of nightly-2022-02-10. We don't refer to this in
// nightly-2022-02-11+ because feature(cfg_target_has_atomic) stabilized.
#[rustfmt::skip]
-static NO_ATOMIC_CAS: &[&str] = &[
+pub(crate) static NO_ATOMIC_CAS: &[&str] = &[
"avr-unknown-gnu-atmega328",
"bpfeb-unknown-none",
"bpfel-unknown-none",
@@ -19,7 +19,7 @@ static NO_ATOMIC_CAS: &[&str] = &[
// Note: This is the list as of nightly-2022-02-10. We don't refer to this in
// nightly-2022-02-11+ because feature(cfg_target_has_atomic) stabilized.
#[rustfmt::skip]
-static NO_ATOMIC_64: &[&str] = &[
+pub(crate) static NO_ATOMIC_64: &[&str] = &[
"arm-linux-androideabi",
"armebv7r-none-eabi",
"armebv7r-none-eabihf",
@@ -69,8 +69,12 @@ static NO_ATOMIC_64: &[&str] = &[
];
#[rustfmt::skip]
-static NO_ATOMIC: &[&str] = &[
+pub(crate) static NO_ATOMIC: &[&str] = &[
+ "armv4t-none-eabi",
+ "armv5te-none-eabi",
"bpfeb-unknown-none",
"bpfel-unknown-none",
"mipsel-sony-psx",
+ "thumbv4t-none-eabi",
+ "thumbv5te-none-eabi",
];
### external/vendor/portable-atomic/src/gen/utils.rs
@@ -2,7 +2,7 @@
// This file is @generated by target_spec.sh.
// It is not intended for manual editing.
-#![allow(unused_macros)]
+#![allow(dead_code, unused_macros)]
// On AArch64, the base register of memory-related instructions must be 64-bit.
// Passing a 32-bit value to `in(reg)` on AArch64 results in the upper bits
@@ -11,25 +11,24 @@
// handle this is to pass it as a pointer and clear the upper bits inside asm,
// but it is easier to overlook than cast, which can catch overlooks by
// asm_sub_register lint.
-// See also https://github.com/ARM-software/abi-aa/blob/2024Q3/aapcs64/aapcs64.rst#pointers
+// See also https://github.com/ARM-software/abi-aa/blob/2025Q1/aapcs64/aapcs64.rst#pointers
//
// Except for x86_64, which can use 32-bit registers in the destination operand
// (on x86_64, we use the ptr_modifier macro to handle this), we need to do the
// same for ILP32 ABI on other 64-bit architectures. (At least, as far as I can
// see from the assembly generated by LLVM, this is also required for MIPS64 N32
-// ABI. I don't know about the RISC-V s64ilp32 ABI for which a patch was
-// recently submitted to the kernel, but in any case, this should be a safe
-// default for such ABIs).
+// ABI. I don't know about the RISC-V RV64ILP32* ABI, but in any case, this
+// should be a safe default for such ABIs).
//
// Known architectures that have such ABI are x86_64 (X32), AArch64 (ILP32),
-// mips64 (N32), and riscv64 (s64ilp32, not merged yet though). (As of
-// 2023-06-05, only the former two are supported by rustc.) However, we list all
-// known 64-bit architectures because similar ABIs may exist or future added for
-// other architectures.
+// mips64 (N32), and riscv64 (RV64ILP32*). (As of 2025-01-23, only the former
+// two are supported by rustc.) However, we list all known 64-bit architectures
+// because similar ABIs may exist or future added for other architectures.
#[cfg(all(
target_pointer_width = "32",
any(
target_arch = "aarch64",
+ target_arch = "amdgpu",
target_arch = "arm64ec",
target_arch = "bpf",
target_arch = "loongarch64",
@@ -44,30 +43,36 @@
target_arch = "x86_64",
),
))]
-#[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
-macro_rules! ptr_reg {
- ($ptr:ident) => {{
- let _: *const _ = $ptr; // ensure $ptr is a pointer (*mut _ or *const _)
- #[cfg(not(portable_atomic_no_asm_maybe_uninit))]
- #[allow(clippy::ptr_as_ptr)]
- {
- // If we cast to u64 here, the provenance will be lost,
- // so we convert to MaybeUninit<u64> via zero extend helper.
- crate::utils::zero_extend64_ptr($ptr as *mut ())
- }
- #[cfg(portable_atomic_no_asm_maybe_uninit)]
- {
- // Use cast on old rustc because it does not support MaybeUninit
- // registers. This is still permissive-provenance compatible and
- // is sound.
- $ptr as u64
- }
- }};
+#[macro_use]
+mod imp {
+ #[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
+ macro_rules! ptr_reg {
+ ($ptr:ident) => {{
+ let _: *const _ = $ptr; // ensure $ptr is a pointer (*mut _ or *const _)
+ #[cfg(not(portable_atomic_no_asm_maybe_uninit))]
+ #[allow(clippy::ptr_as_ptr)]
+ {
+ // If we cast to u64 here, the provenance will be lost,
+ // so we convert to MaybeUninit<u64> via zero extend helper.
+ crate::utils::zero_extend64_ptr($ptr as *mut ())
+ }
+ #[cfg(portable_atomic_no_asm_maybe_uninit)]
+ {
+ // Use cast on old rustc because it does not support MaybeUninit
+ // registers. This is still permissive-provenance compatible and
+ // is sound.
+ $ptr as u64
+ }
+ }};
+ }
+ pub(crate) type RegISize = i64;
+ pub(crate) type RegSize = u64;
}
#[cfg(not(all(
target_pointer_width = "32",
any(
target_arch = "aarch64",
+ target_arch = "amdgpu",
target_arch = "arm64ec",
target_arch = "bpf",
target_arch = "loongarch64",
@@ -82,13 +87,19 @@ macro_rules! ptr_reg {
target_arch = "x86_64",
),
)))]
-#[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
-macro_rules! ptr_reg {
- ($ptr:ident) => {{
- let _: *const _ = $ptr; // ensure $ptr is a pointer (*mut _ or *const _)
- $ptr // cast is unnecessary here.
- }};
+#[macro_use]
+mod imp {
+ #[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
+ macro_rules! ptr_reg {
+ ($ptr:ident) => {{
+ let _: *const _ = $ptr; // ensure $ptr is a pointer (*mut _ or *const _)
+ $ptr // cast is unnecessary here.
+ }};
+ }
+ pub(crate) type RegISize = isize;
+ pub(crate) type RegSize = usize;
}
+pub(crate) use self::imp::{RegISize, RegSize};
// Some 64-bit architectures have ABI with 32-bit pointer width (e.g., x86_64 X32 ABI,
// AArch64 ILP32 ABI, MIPS64 N32 ABI). On those targets, AtomicU64 is available
@@ -98,6 +109,7 @@ macro_rules! ptr_reg {
#[cfg(any(
not(any(target_pointer_width = "16", target_pointer_width = "32")), // i.e., 64-bit or greater
target_arch = "aarch64",
+ target_arch = "amdgpu",
target_arch = "arm64ec",
target_arch = "bpf",
target_arch = "loongarch64",
@@ -125,6 +137,7 @@ mod fast_atomic_64_macros {
#[cfg(not(any(
not(any(target_pointer_width = "16", target_pointer_width = "32")), // i.e., 64-bit or greater
target_arch = "aarch64",
+ target_arch = "amdgpu",
target_arch = "arm64ec",
target_arch = "bpf",
target_arch = "loongarch64",
### external/vendor/portable-atomic/src/imp/atomic128/README.md
@@ -8,11 +8,14 @@ Here is the table of targets that support 128-bit atomics and the instructions u
| target_arch | load | store | CAS | RMW | note |
| ----------- | ---- | ----- | --- | --- | ---- |
-| x86_64 | cmpxchg16b or vmovdqa | cmpxchg16b or vmovdqa | cmpxchg16b | cmpxchg16b | Requires `cmpxchg16b` target feature (enabled by default on Apple and Windows (except Windows 7) targets). vmovdqa requires Intel, AMD, or Zhaoxin CPU with AVX. <br> Both compile-time and run-time detection are supported for cmpxchg16b. vmovdqa is currently run-time detection only. <br> Requires rustc 1.59+ |
-| aarch64/arm64ec | ldxp/stxp or casp or ldp/ldiapp | ldxp/stxp or casp or stp/stilp/swpp | ldxp/stxp or casp | ldxp/stxp or casp/swpp/ldclrp/ldsetp | casp requires `lse` target feature, ldp/stp requires `lse2` target feature, ldiapp/stilp requires `lse2` and `rcpc3` target features, swpp/ldclrp/ldsetp requires `lse128` target feature. <br> Both compile-time and run-time detection are supported. <br> Requires rustc 1.59+ (aarch64) / 1.84+ (arm64ec) |
-| riscv64 | amocas.q | amocas.q | amocas.q | amocas.q | Experimental because LLVM marking the corresponding target feature as experimental. Requires `experimental-zacas` target feature. Both compile-time and run-time detection are supported (run-time detection is currently disabled by default). <br> Requires rustc 1.59+ |
-| powerpc64 | lq | stq | lqarx/stqcx. | lqarx/stqcx. | Requires `quadword-atomics` target feature (enabled by default on powerpc64le). Both compile-time and run-time detection are supported. <br> Requires nightly |
-| s390x | lpq | stpq | cdsg | cdsg | Requires rustc 1.84+ |
+| x86_64 | cmpxchg16b or vmovdqa | cmpxchg16b or vmovdqa | cmpxchg16b | cmpxchg16b | Requires `cmpxchg16b` target feature (enabled by default on Apple, Windows (except Windows 7, since Rust 1.78), and Fuchsia (since Rust 1.87) targets). vmovdqa requires `avx` target feature. <br> Both compile-time and run-time detection are supported. <br> Requires Rust 1.59+ |
+| aarch64/arm64ec | ldxp/stxp or casp or ldp/ldiapp | ldxp/stxp or casp or stp/stilp/swpp | ldxp/stxp or casp | ldxp/stxp or casp/swpp/ldclrp/ldsetp | casp requires `lse` target feature, ldp/stp requires `lse2` target feature, ldiapp/stilp requires `lse2` and `rcpc3` target features, swpp/ldclrp/ldsetp requires `lse128` target feature. <br> Both compile-time and run-time detection are supported. <br> Requires Rust 1.59+ (aarch64) / 1.84+ (arm64ec) |
+| riscv64 | amocas.q | amocas.q | amocas.q | amocas.q | Requires `zacas` target feature. Both compile-time and run-time detection are supported. <br> Requires Rust 1.59+ |
+| powerpc64 | lq | stq | lqarx/stqcx. | lqarx/stqcx. | Requires `quadword-atomics` target feature (enabled by default on powerpc64le). Both compile-time and run-time detection are supported. <br> Requires Rust 1.95+ |
+| s390x | lpq | stpq | cdsg | cdsg | Requires Rust 1.84+ |
+| loongarch64 | sc.q | sc.q | sc.q | sc.q | Unimplemented. Requires `scq` target feature. |
+| mips64r6 | lldp | lldp/scdp | lldp/scdp | lldp/scdp | Unimplemented (unsupported in LLVM). Requires Release 6 Paired LL/SC family of instructions |
+| nvptx64 | ld.b128 | st.b128 | atom.cas.b128 | atom.exch.b128/atom.cas.b128 | Unimplemented. Requires `ptx83` and `sm_90`. |
On compiler versions or platforms where these are not supported, the fallback implementation is used.
### external/vendor/portable-atomic/src/imp/atomic128/aarch64.rs
@@ -3,13 +3,17 @@
/*
128-bit atomic implementation on AArch64.
-There are a few ways to implement 128-bit atomic operations in AArch64.
+This architecture provides the following 128-bit atomic instructions:
-- LDXP/STXP loop (DW LL/SC)
-- CASP (DWCAS) added as Armv8.1 FEAT_LSE (optional from Armv8.0, mandatory from Armv8.1)
-- LDP/STP (DW load/store) if Armv8.4 FEAT_LSE2 (optional from Armv8.2, mandatory from Armv8.4) is available
-- LDIAPP/STILP (DW acquire-load/release-store) added as Armv8.9 FEAT_LRCPC3 (optional from Armv8.2) (if FEAT_LSE2 is also available)
-- LDCLRP/LDSETP/SWPP (DW RMW) added as Armv9.4 FEAT_LSE128 (optional from Armv9.3)
+- LDXP/STXP: LL/SC (Armv8.0 baseline)
+- CASP: CAS (added as Armv8.1 FEAT_LSE (optional from Armv8.0, mandatory from Armv8.1))
+- LDP/STP: load/store (if Armv8.4 FEAT_LSE2 (optional from Armv8.2, mandatory from Armv8.4) is available)
+- LDIAPP/STILP: acquire-load/release-store (added as Armv8.9 FEAT_LRCPC3 (optional from Armv8.2) (if FEAT_LSE2 is also available))
+- LDCLRP/LDSETP/SWPP: fetch-and-{clear,or},swap (added as Armv9.4 FEAT_LSE128 (optional from Armv9.3))
+
+See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
+detailed description of the atomic and synchronize instructions in this architecture:
+https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#aarch64
This module supports all of these instructions and attempts to select the best
one based on compile-time and run-time information about available CPU features
@@ -59,22 +63,18 @@ this module and use intrinsics.rs instead.
Refs:
- Arm A-profile A64 Instruction Set Architecture
- https://developer.arm.com/documentation/ddi0602/2024-06
+ https://developer.arm.com/documentation/ddi0602/2025-06
+- C/C++ Atomics Application Binary Interface Standard for the Arm® 64-bit Architecture
+ https://github.com/ARM-software/abi-aa/blob/2025Q1/atomicsabi64/atomicsabi64.rst
- Arm Compiler armasm User Guide
https://developer.arm.com/documentation/dui0801/latest
- Arm Architecture Reference Manual for A-profile architecture
https://developer.arm.com/documentation/ddi0487/latest (PDF)
+- Arm Architecture Reference Manual Supplement - Armv8, for Armv8-R AArch64 architecture profile
+ https://developer.arm.com/documentation/ddi0600/latest (PDF)
- atomic-maybe-uninit https://github.com/taiki-e/atomic-maybe-uninit
-Generated asm:
-- aarch64 https://godbolt.org/z/aEWe7zhMh
-- aarch64 msvc https://godbolt.org/z/Phq7M6MPs
-- aarch64 (+lse) https://godbolt.org/z/9Go3dT6sW
-- aarch64 msvc (+lse) https://godbolt.org/z/vGvc6bTMT
-- aarch64 (+lse,+lse2) https://godbolt.org/z/KddzqsM9o
-- aarch64 (+lse,+lse2,+rcpc3) https://godbolt.org/z/sePheahxh
-- aarch64 (+lse2,+lse128) https://godbolt.org/z/WPqM9M1r3
-- aarch64 (+lse2,+lse128,+rcpc3) https://godbolt.org/z/5Mf8dc88Y
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
include!("macros.rs");
@@ -94,10 +94,9 @@ include!("macros.rs");
target_os = "linux",
any(
target_env = "gnu",
- all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
- ),
+ all(target_env = "musl", any(not(target_feature = "crt-static"), feature = "std")),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
@@ -136,28 +135,26 @@ mod detect;
mod detect;
#[cfg(not(portable_atomic_no_outline_atomics))]
#[cfg(any(test, not(any(target_feature = "lse", portable_atomic_target_feature = "lse"))))]
-#[cfg(target_os = "windows")]
+#[cfg(windows)]
#[path = "../detect/aarch64_windows.rs"]
mod detect;
-// test only
-#[cfg(test)]
-#[cfg(not(qemu))]
+#[cfg(test)] // test-only (we use auxv.rs)
#[cfg(not(valgrind))]
#[cfg(not(portable_atomic_no_outline_atomics))]
#[cfg(any(target_os = "linux", target_os = "android", target_os = "freebsd"))]
#[path = "../detect/aarch64_aa64reg.rs"]
-mod detect_aa64reg;
-#[cfg(test)]
+mod test_detect_aa64reg;
+#[cfg(test)] // test-only (unused)
#[cfg(not(portable_atomic_no_outline_atomics))]
#[cfg(target_vendor = "apple")]
#[path = "../detect/aarch64_apple.rs"]
-mod detect_apple;
-#[cfg(test)]
+mod test_detect_apple;
+#[cfg(test)] // test-only (we use aarch64_aa64reg.rs)
#[cfg(not(portable_atomic_no_outline_atomics))]
#[cfg(target_os = "openbsd")]
#[path = "../detect/auxv.rs"]
-mod detect_auxv;
+mod test_detect_auxv;
#[cfg(not(portable_atomic_no_asm))]
use core::arch::asm;
@@ -170,6 +167,7 @@ use crate::utils::{Pair, U128};
portable_atomic_target_feature = "lse",
not(portable_atomic_no_outline_atomics),
))]
+#[rustfmt::skip]
macro_rules! debug_assert_lse {
() => {
#[cfg(all(
@@ -180,9 +178,11 @@ macro_rules! debug_assert_lse {
any(
target_env = "gnu",
all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
@@ -192,21 +192,21 @@ macro_rules! debug_assert_lse {
target_os = "openbsd",
all(target_os = "illumos", portable_atomic_outline_atomics),
target_os = "fuchsia",
- target_os = "windows",
+ windows,
),
))]
#[cfg(not(any(target_feature = "lse", portable_atomic_target_feature = "lse")))]
{
- debug_assert!(detect::detect().has_lse());
+ debug_assert!(detect::detect().lse());
}
};
}
-#[rustfmt::skip]
#[cfg(any(
target_feature = "lse2",
portable_atomic_target_feature = "lse2",
not(portable_atomic_no_outline_atomics),
))]
+#[rustfmt::skip]
macro_rules! debug_assert_lse2 {
() => {
#[cfg(all(
@@ -217,9 +217,11 @@ macro_rules! debug_assert_lse2 {
any(
target_env = "gnu",
all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
@@ -230,16 +232,15 @@ macro_rules! debug_assert_lse2 {
all(target_os = "illumos", portable_atomic_outline_atomics),
// These don't support detection of FEAT_LSE2.
// target_os = "fuchsia",
- // target_os = "windows",
+ // windows,
),
))]
#[cfg(not(any(target_feature = "lse2", portable_atomic_target_feature = "lse2")))]
{
- debug_assert!(detect::detect().has_lse2());
+ debug_assert!(detect::detect().lse2());
}
};
}
-#[rustfmt::skip]
#[cfg(any(
target_feature = "lse128",
portable_atomic_target_feature = "lse128",
@@ -248,6 +249,7 @@ macro_rules! debug_assert_lse2 {
not(any(target_feature = "lse2", portable_atomic_target_feature = "lse2")),
),
))]
+#[rustfmt::skip]
macro_rules! debug_assert_lse128 {
() => {
#[cfg(all(
@@ -258,9 +260,11 @@ macro_rules! debug_assert_lse128 {
any(
target_env = "gnu",
all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
@@ -271,16 +275,15 @@ macro_rules! debug_assert_lse128 {
all(target_os = "illumos", portable_atomic_outline_atomics),
// These don't support detection of FEAT_LSE128.
// target_os = "fuchsia",
- // target_os = "windows",
+ // windows,
),
))]
#[cfg(not(any(target_feature = "lse128", portable_atomic_target_feature = "lse128")))]
{
- debug_assert!(detect::detect().has_lse128());
+ debug_assert!(detect::detect().lse128());
}
};
}
-#[rustfmt::skip]
#[cfg(any(
target_feature = "rcpc3",
portable_atomic_target_feature = "rcpc3",
@@ -289,6 +292,7 @@ macro_rules! debug_assert_lse128 {
not(any(target_feature = "lse2", portable_atomic_target_feature = "lse2")),
),
))]
+#[rustfmt::skip]
macro_rules! debug_assert_rcpc3 {
() => {
#[cfg(all(
@@ -299,9 +303,11 @@ macro_rules! debug_assert_rcpc3 {
any(
target_env = "gnu",
all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
@@ -312,12 +318,12 @@ macro_rules! debug_assert_rcpc3 {
all(target_os = "illumos", portable_atomic_outline_atomics),
// These don't support detection of FEAT_LRCPC3.
// target_os = "fuchsia",
- // target_os = "windows",
+ // windows,
),
))]
#[cfg(not(any(target_feature = "rcpc3", portable_atomic_target_feature = "rcpc3")))]
{
- debug_assert!(detect::detect().has_rcpc3());
+ debug_assert!(detect::detect().rcpc3());
}
};
}
@@ -326,22 +332,24 @@ macro_rules! debug_assert_rcpc3 {
//
// This is similar to #[target_feature(enable = "lse")], except that there are
// no compiler guarantees regarding (un)inlining, and the scope is within an asm
-// block rather than a function. We use this directive to support outline-atomics
-// on pre-1.61 rustc (aarch64_target_feature stabilized in Rust 1.61).
+// block rather than a function. We use this directive because #[target_feature(enable = "lse")]
+// is unstable on pre-1.61 rustc and incompatible with rustc_codegen_cranelift:
+// https://github.com/rust-lang/rustc_codegen_cranelift/issues/1400#issuecomment-1774599775
//
-// The .arch_extension directive is effective until the end of the assembly block and
+// The .arch_extension directive in asm! is effective until the end of the assembly block and
// is not propagated to subsequent code, so the end_lse macro is unneeded.
// https://godbolt.org/z/o6EPndP94
// https://github.com/torvalds/linux/commit/e0d5896bd356cd577f9710a02d7a474cdf58426b
// https://github.com/torvalds/linux/commit/dd1f6308b28edf0452dd5dc7877992903ec61e69
// (It seems GCC effectively ignores this directive and always allow FEAT_LSE instructions: https://godbolt.org/z/W9W6rensG)
+// Note that the .arch_extension directive in global_asm!/naked_asm! which are
+// not used in this crate has different behavior: https://github.com/rust-lang/rust/pull/137720#discussion_r1973608259
+// Note that this directive currently cannot be used correctly with global_asm!/naked_asm!
+// due to LLVM bug: https://github.com/rust-lang/rust/pull/137720#discussion_r2014505753
//
// The .arch directive has a similar effect, but we don't use it due to the following issue:
// https://github.com/torvalds/linux/commit/dd1f6308b28edf0452dd5dc7877992903ec61e69
//
-// This is also needed for compatibility with rustc_codegen_cranelift:
-// https://github.com/rust-lang/rustc_codegen_cranelift/issues/1400#issuecomment-1774599775
-//
// Note: If FEAT_LSE is not available at compile-time, we must guarantee that
// the function that uses it is not inlined into a function where it is not
// clear whether FEAT_LSE is available. Otherwise, (even if we checked whether
@@ -437,10 +445,10 @@ macro_rules! atomic_rmw_inst {
};
($op:ident, $order:ident, write = $write:ident) => {
match $order {
- Ordering::Relaxed => $op!("2", ""),
- Ordering::Acquire => $op!("a", ""),
- Ordering::Release => $op!("6", ""),
- Ordering::AcqRel => $op!("e", ""),
+ Ordering::Relaxed => $op!("2", ""), // ""
+ Ordering::Acquire => $op!("a", ""), // "a"
+ Ordering::Release => $op!("6", ""), // "l"
+ Ordering::AcqRel => $op!("e", ""), // "al"
// In MSVC environments, SeqCst stores/writes needs fences after writes.
// https://reviews.llvm.org/D141748
#[cfg(target_env = "msvc")]
@@ -480,28 +488,12 @@ Note:
*/
-// if compile_time(FEAT_LSE2) => ldp:
-// cfg guarantee that the CPU supports FEAT_LSE2.
-#[cfg(any(target_feature = "lse2", portable_atomic_target_feature = "lse2"))]
-use self::_atomic_load_ldp as atomic_load;
-#[cfg(not(any(target_feature = "lse2", portable_atomic_target_feature = "lse2")))]
-#[inline]
-unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
- #[inline]
- unsafe fn atomic_load_no_lse2(src: *mut u128, order: Ordering) -> u128 {
- // if compile_time(FEAT_LSE) => casp
- #[cfg(any(target_feature = "lse", portable_atomic_target_feature = "lse"))]
- // SAFETY: the caller must uphold the safety contract.
- // cfg guarantee that the CPU supports FEAT_LSE.
- unsafe {
- _atomic_load_casp(src, order)
- }
- // else => ldxp_stxp
- #[cfg(not(any(target_feature = "lse", portable_atomic_target_feature = "lse")))]
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- _atomic_load_ldxp_stxp(src, order)
- }
+cfg_sel!({
+ // if compile_time(FEAT_LSE2) => ldp:
+ // cfg guarantee that the CPU supports FEAT_LSE2.
+ #[cfg(any(target_feature = "lse2", portable_atomic_target_feature = "lse2"))]
+ {
+ use self::_atomic_load_ldp as atomic_load;
}
// if platform_supports_detection_of(FEAT_LSE2):
#[cfg(all(
@@ -512,9 +504,11 @@ unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
any(
target_env = "gnu",
all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
@@ -525,111 +519,104 @@ unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
all(target_os = "illumos", portable_atomic_outline_atomics),
// These don't support detection of FEAT_LSE2.
// target_os = "fuchsia",
- // target_os = "windows",
+ // windows,
),
))]
{
- fn_alias! {
- // inline(never) is just a hint and also not strictly necessary
- // because we use ifunc helper macro, but used for clarity.
- #[inline(never)]
- unsafe fn(src: *mut u128) -> u128;
- atomic_load_lse2_relaxed = _atomic_load_ldp(Ordering::Relaxed);
- atomic_load_lse2_acquire = _atomic_load_ldp(Ordering::Acquire);
- atomic_load_lse2_seqcst = _atomic_load_ldp(Ordering::SeqCst);
- atomic_load_lse2_rcpc3_acquire = _atomic_load_ldiapp(Ordering::Acquire);
- atomic_load_lse2_rcpc3_seqcst = _atomic_load_ldiapp(Ordering::SeqCst);
- }
- fn_alias! {
- unsafe fn(src: *mut u128) -> u128;
- atomic_load_no_lse2_relaxed = atomic_load_no_lse2(Ordering::Relaxed);
- atomic_load_no_lse2_acquire = atomic_load_no_lse2(Ordering::Acquire);
- atomic_load_no_lse2_seqcst = atomic_load_no_lse2(Ordering::SeqCst);
- }
- // SAFETY: the caller must uphold the safety contract.
- // and we've checked if FEAT_LSE2 is available.
- unsafe {
- match order {
- Ordering::Relaxed => {
- ifunc!(unsafe fn(src: *mut u128) -> u128 {
- let cpuinfo = detect::detect();
- if cpuinfo.has_lse2() {
- // if detect(FEAT_LSE2) => lse2 (ldp)
- atomic_load_lse2_relaxed
- } else {
- // else => no_lse2:
- atomic_load_no_lse2_relaxed
- }
- })
- }
- Ordering::Acquire => {
- ifunc!(unsafe fn(src: *mut u128) -> u128 {
- let cpuinfo = detect::detect();
- if cpuinfo.has_lse2() {
- if cpuinfo.has_rcpc3() {
- // if detect(FEAT_LSE2) && detect(FEAT_LRCPC3) && order != relaxed => lse2_rcpc3 (ldiapp)
- atomic_load_lse2_rcpc3_acquire
- } else {
+ #[inline]
+ unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
+ fn_alias! {
+ // inline(never) is just a hint and also not strictly necessary
+ // because we use ifunc helper macro, but used for clarity.
+ #[inline(never)]
+ unsafe fn(src: *mut u128) -> u128;
+ atomic_load_lse2_relaxed = _atomic_load_ldp(Ordering::Relaxed);
+ atomic_load_lse2_acquire = _atomic_load_ldp(Ordering::Acquire);
+ atomic_load_lse2_seqcst = _atomic_load_ldp(Ordering::SeqCst);
+ atomic_load_lse2_rcpc3_acquire = _atomic_load_ldiapp(Ordering::Acquire);
+ atomic_load_lse2_rcpc3_seqcst = _atomic_load_ldiapp(Ordering::SeqCst);
+ }
+ fn_alias! {
+ unsafe fn(src: *mut u128) -> u128;
+ atomic_load_no_lse2_relaxed = atomic_load_no_lse2(Ordering::Relaxed);
+ atomic_load_no_lse2_acquire = atomic_load_no_lse2(Ordering::Acquire);
+ atomic_load_no_lse2_seqcst = atomic_load_no_lse2(Ordering::SeqCst);
+ }
+ // SAFETY: the caller must uphold the safety contract.
+ // and we've checked if FEAT_LSE2/FEAT_LRCPC3 is available.
+ unsafe {
+ match order {
+ Ordering::Relaxed => {
+ ifunc!(unsafe fn(src: *mut u128) -> u128 {
+ let cpuinfo = detect::detect();
+ if cpuinfo.lse2() {
// if detect(FEAT_LSE2) => lse2 (ldp)
- atomic_load_lse2_acquire
+ atomic_load_lse2_relaxed
+ } else {
+ // else => no_lse2:
+ atomic_load_no_lse2_relaxed
}
- } else {
- // else => no_lse2:
- atomic_load_no_lse2_acquire
- }
- })
- }
- Ordering::SeqCst => {
- ifunc!(unsafe fn(src: *mut u128) -> u128 {
- let cpuinfo = detect::detect();
- if cpuinfo.has_lse2() {
- if cpuinfo.has_rcpc3() {
- // if detect(FEAT_LSE2) && detect(FEAT_LRCPC3) && order != relaxed => lse2_rcpc3 (ldiapp)
- atomic_load_lse2_rcpc3_seqcst
+ })
+ }
+ Ordering::Acquire => {
+ ifunc!(unsafe fn(src: *mut u128) -> u128 {
+ let cpuinfo = detect::detect();
+ if cpuinfo.lse2() {
+ if cpuinfo.rcpc3() {
+ // if detect(FEAT_LSE2) && detect(FEAT_LRCPC3) && order != relaxed => lse2_rcpc3 (ldiapp)
+ atomic_load_lse2_rcpc3_acquire
+ } else {
+ // if detect(FEAT_LSE2) => lse2 (ldp)
+ atomic_load_lse2_acquire
+ }
} else {
- // if detect(FEAT_LSE2) => lse2 (ldp)
- atomic_load_lse2_seqcst
+ // else => no_lse2:
+ atomic_load_no_lse2_acquire
}
- } else {
- // else => no_lse2:
- atomic_load_no_lse2_seqcst
- }
- })
+ })
+ }
+ Ordering::SeqCst => {
+ ifunc!(unsafe fn(src: *mut u128) -> u128 {
+ let cpuinfo = detect::detect();
+ if cpuinfo.lse2() {
+ if cpuinfo.rcpc3() {
+ // if detect(FEAT_LSE2) && detect(FEAT_LRCPC3) && order != relaxed => lse2_rcpc3 (ldiapp)
+ atomic_load_lse2_rcpc3_seqcst
+ } else {
+ // if detect(FEAT_LSE2) => lse2 (ldp)
+ atomic_load_lse2_seqcst
+ }
+ } else {
+ // else => no_lse2:
+ atomic_load_no_lse2_seqcst
+ }
+ })
+ }
+ _ => unreachable!(),
}
- _ => unreachable!(),
}
}
}
// else => no_lse2:
- #[cfg(not(all(
- not(portable_atomic_no_outline_atomics),
- any(
- all(
- target_os = "linux",
- any(
- target_env = "gnu",
- all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
- ),
- portable_atomic_outline_atomics,
- ),
- ),
- target_os = "android",
- target_os = "freebsd",
- target_os = "netbsd",
- target_os = "openbsd",
- all(target_os = "illumos", portable_atomic_outline_atomics),
- // These don't support detection of FEAT_LSE2.
- // target_os = "fuchsia",
- // target_os = "windows",
- ),
- )))]
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- atomic_load_no_lse2(src, order)
+ #[cfg(else)]
+ {
+ use self::atomic_load_no_lse2 as atomic_load;
}
-}
+});
+#[cfg(not(any(target_feature = "lse2", portable_atomic_target_feature = "lse2")))]
+cfg_sel!({
+ // if compile_time(FEAT_LSE) => casp
+ // cfg guarantee that the CPU supports FEAT_LSE.
+ #[cfg(any(target_feature = "lse", portable_atomic_target_feature = "lse"))]
+ {
+ use self::_atomic_load_casp as atomic_load_no_lse2;
+ }
+ // else => ldxp_stxp
+ #[cfg(else)]
+ {
+ use self::_atomic_load_ldxp_stxp as atomic_load_no_lse2;
+ }
+});
// If CPU supports FEAT_LSE2, LDP/LDIAPP is single-copy atomic reads,
// otherwise it is two single-copy atomic reads.
// Refs: B2.2.1 of the Arm Architecture Reference Manual Armv8, for Armv8-A architecture profile
@@ -646,7 +633,7 @@ unsafe fn _atomic_load_ldp(src: *mut u128, order: Ordering) -> u128 {
// SAFETY: the caller must guarantee that `dst` is valid for reads,
// 16-byte aligned, that there are no concurrent non-atomic operations.
//
- // Refs: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/LDP--Load-pair-of-registers-
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/LDP--Load-pair-of-registers-
unsafe {
let (out_lo, out_hi);
macro_rules! atomic_load_relaxed {
@@ -714,7 +701,7 @@ unsafe fn _atomic_load_ldiapp(src: *mut u128, order: Ordering) -> u128 {
// SAFETY: the caller must guarantee that `dst` is valid for reads,
// 16-byte aligned, that there are no concurrent non-atomic operations.
//
- // Refs: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/LDIAPP--Load-Acquire-RCpc-ordered-pair-of-registers-
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/LDIAPP--Load-Acquire-RCpc-ordered-pair-of-registers-
unsafe {
let (out_lo, out_hi);
match order {
@@ -732,7 +719,7 @@ unsafe fn _atomic_load_ldiapp(src: *mut u128, order: Ordering) -> u128 {
// https://github.com/llvm/llvm-project/commit/a6aaa969f7caec58a994142f8d855861cf3a1463
#[cfg(portable_atomic_pre_llvm_16)]
asm!(
- // 0: d9411800 ldiapp x0, x1, [x0]
+ // ldiapp x0, x1, [x0]
".inst 0xd9411800",
in("x0") ptr_reg!(src),
lateout("x1") out_hi,
@@ -761,7 +748,7 @@ unsafe fn _atomic_load_ldiapp(src: *mut u128, order: Ordering) -> u128 {
// ldar (or dmb ishld) is required to prevent reordering with preceding stlxp.
// See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108891 for details.
"ldar {tmp}, [x0]",
- // 0: d9411800 ldiapp x0, x1, [x0]
+ // ldiapp x0, x1, [x0]
".inst 0xd9411800",
tmp = out(reg) _,
in("x0") ptr_reg!(src),
@@ -880,51 +867,12 @@ Note:
*/
-// if compile_time(FEAT_LSE2) => stp:
-// cfg guarantee that the CPU supports FEAT_LSE2.
-#[cfg(any(target_feature = "lse2", portable_atomic_target_feature = "lse2"))]
-use self::_atomic_store_stp as atomic_store;
-#[cfg(not(any(target_feature = "lse2", portable_atomic_target_feature = "lse2")))]
-#[inline]
-unsafe fn atomic_store(dst: *mut u128, val: u128, order: Ordering) {
- #[inline]
- unsafe fn atomic_store_no_lse2(dst: *mut u128, val: u128, order: Ordering) {
- // if compile_time(FEAT_LSE) && not(ll_sc_rmw) => casp
- // If FEAT_LSE is available at compile-time and portable_atomic_ll_sc_rmw cfg is not set,
- // we use CAS-based atomic RMW.
- #[cfg(all(
- any(target_feature = "lse", portable_atomic_target_feature = "lse"),
- not(portable_atomic_ll_sc_rmw),
- ))]
- // SAFETY: the caller must uphold the safety contract.
- // cfg guarantee that the CPU supports FEAT_LSE.
- unsafe {
- _atomic_swap_casp(dst, val, order);
- }
- // else => ldxp_stxp
- #[cfg(not(all(
- any(target_feature = "lse", portable_atomic_target_feature = "lse"),
- not(portable_atomic_ll_sc_rmw),
- )))]
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- _atomic_store_ldxp_stxp(dst, val, order);
- }
- }
- #[cfg(any(
- target_feature = "lse128",
- portable_atomic_target_feature = "lse128",
- all(
- not(portable_atomic_no_outline_atomics),
- not(any(target_feature = "lse2", portable_atomic_target_feature = "lse2")),
- ),
- ))]
- #[inline]
- unsafe fn _atomic_store_swpp(dst: *mut u128, val: u128, order: Ordering) {
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- _atomic_swap_swpp(dst, val, order);
- }
+cfg_sel!({
+ // if compile_time(FEAT_LSE2) => stp:
+ // cfg guarantee that the CPU supports FEAT_LSE2.
+ #[cfg(any(target_feature = "lse2", portable_atomic_target_feature = "lse2"))]
+ {
+ use self::_atomic_store_stp as atomic_store;
}
// if platform_supports_detection_of(FEAT_LSE2):
#[cfg(all(
@@ -935,9 +883,11 @@ unsafe fn atomic_store(dst: *mut u128, val: u128, order: Ordering) {
any(
target_env = "gnu",
all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
@@ -948,119 +898,138 @@ unsafe fn atomic_store(dst: *mut u128, val: u128, order: Ordering) {
all(target_os = "illumos", portable_atomic_outline_atomics),
// These don't support detection of FEAT_LSE2.
// target_os = "fuchsia",
- // target_os = "windows",
+ // windows,
),
))]
{
- fn_alias! {
- // inline(never) is just a hint and also not strictly necessary
- // because we use ifunc helper macro, but used for clarity.
- #[inline(never)]
- unsafe fn(dst: *mut u128, val: u128);
- atomic_store_lse2_relaxed = _atomic_store_stp(Ordering::Relaxed);
- atomic_store_lse2_release = _atomic_store_stp(Ordering::Release);
- atomic_store_lse2_seqcst = _atomic_store_stp(Ordering::SeqCst);
- atomic_store_lse2_rcpc3_release = _atomic_store_stilp(Ordering::Release);
- atomic_store_lse2_rcpc3_seqcst = _atomic_store_stilp(Ordering::SeqCst);
- atomic_store_lse128_release = _atomic_store_swpp(Ordering::Release);
- atomic_store_lse128_seqcst = _atomic_store_swpp(Ordering::SeqCst);
- }
- fn_alias! {
- unsafe fn(dst: *mut u128, val: u128);
- atomic_store_no_lse2_relaxed = atomic_store_no_lse2(Ordering::Relaxed);
- atomic_store_no_lse2_release = atomic_store_no_lse2(Ordering::Release);
- atomic_store_no_lse2_seqcst = atomic_store_no_lse2(Ordering::SeqCst);
- }
- // SAFETY: the caller must uphold the safety contract.
- // and we've checked if FEAT_LSE2 is available.
- unsafe {
- match order {
- Ordering::Relaxed => {
- ifunc!(unsafe fn(dst: *mut u128, val: u128) {
- let cpuinfo = detect::detect();
- if cpuinfo.has_lse2() {
- // if detect(FEAT_LSE2) => lse2 (stp)
- atomic_store_lse2_relaxed
- } else {
- // else => no_lse2:
- atomic_store_no_lse2_relaxed
- }
- });
+ #[inline]
+ unsafe fn atomic_store(dst: *mut u128, val: u128, order: Ordering) {
+ #[cfg(any(
+ target_feature = "lse128",
+ portable_atomic_target_feature = "lse128",
+ all(
+ not(portable_atomic_no_outline_atomics),
+ not(any(target_feature = "lse2", portable_atomic_target_feature = "lse2")),
+ ),
+ ))]
+ #[inline]
+ unsafe fn _atomic_store_swpp(dst: *mut u128, val: u128, order: Ordering) {
+ // SAFETY: the caller must uphold the safety contract.
+ unsafe {
+ _atomic_swap_swpp(dst, val, order);
}
- Ordering::Release => {
- ifunc!(unsafe fn(dst: *mut u128, val: u128) {
- let cpuinfo = detect::detect();
- if cpuinfo.has_lse2() {
- if cpuinfo.has_rcpc3() {
- // if detect(FEAT_LSE2) && detect(FEAT_LRCPC3) && order != relaxed => lse2_rcpc3 (stilp)
- atomic_store_lse2_rcpc3_release
- } else if cpuinfo.has_lse128() {
- // if detect(FEAT_LSE2) && detect(FEAT_LSE128) && order != relaxed => lse128 (swpp)
- atomic_store_lse128_release
- } else {
+ }
+ fn_alias! {
+ // inline(never) is just a hint and also not strictly necessary
+ // because we use ifunc helper macro, but used for clarity.
+ #[inline(never)]
+ unsafe fn(dst: *mut u128, val: u128);
+ atomic_store_lse2_relaxed = _atomic_store_stp(Ordering::Relaxed);
+ atomic_store_lse2_release = _atomic_store_stp(Ordering::Release);
+ atomic_store_lse2_seqcst = _atomic_store_stp(Ordering::SeqCst);
+ atomic_store_lse2_rcpc3_release = _atomic_store_stilp(Ordering::Release);
+ atomic_store_lse2_rcpc3_seqcst = _atomic_store_stilp(Ordering::SeqCst);
+ atomic_store_lse128_release = _atomic_store_swpp(Ordering::Release);
+ atomic_store_lse128_seqcst = _atomic_store_swpp(Ordering::SeqCst);
+ }
+ fn_alias! {
+ unsafe fn(dst: *mut u128, val: u128);
+ atomic_store_no_lse2_relaxed = atomic_store_no_lse2(Ordering::Relaxed);
+ atomic_store_no_lse2_release = atomic_store_no_lse2(Ordering::Release);
+ atomic_store_no_lse2_seqcst = atomic_store_no_lse2(Ordering::SeqCst);
+ }
+ // SAFETY: the caller must uphold the safety contract.
+ // and we've checked if FEAT_LSE2/FEAT_LRCPC3/FEAT_LSE128 is available.
+ unsafe {
+ match order {
+ Ordering::Relaxed => {
+ ifunc!(unsafe fn(dst: *mut u128, val: u128) {
+ let cpuinfo = detect::detect();
+ if cpuinfo.lse2() {
// if detect(FEAT_LSE2) => lse2 (stp)
- atomic_store_lse2_release
+ atomic_store_lse2_relaxed
+ } else {
+ // else => no_lse2:
+ atomic_store_no_lse2_relaxed
}
- } else {
- // else => no_lse2:
- atomic_store_no_lse2_release
- }
- });
- }
- Ordering::SeqCst => {
- ifunc!(unsafe fn(dst: *mut u128, val: u128) {
- let cpuinfo = detect::detect();
- if cpuinfo.has_lse2() {
- if cpuinfo.has_lse128() {
- // if detect(FEAT_LSE2) && detect(FEAT_LSE128) && order == seqcst => lse128 (swpp)
- atomic_store_lse128_seqcst
- } else if cpuinfo.has_rcpc3() {
- // if detect(FEAT_LSE2) && detect(FEAT_LRCPC3) && order != relaxed => lse2_rcpc3 (stilp)
- atomic_store_lse2_rcpc3_seqcst
+ });
+ }
+ Ordering::Release => {
+ ifunc!(unsafe fn(dst: *mut u128, val: u128) {
+ let cpuinfo = detect::detect();
+ if cpuinfo.lse2() {
+ if cpuinfo.rcpc3() {
+ // if detect(FEAT_LSE2) && detect(FEAT_LRCPC3) && order != relaxed => lse2_rcpc3 (stilp)
+ atomic_store_lse2_rcpc3_release
+ } else if cpuinfo.lse128() {
+ // if detect(FEAT_LSE2) && detect(FEAT_LSE128) && order != relaxed => lse128 (swpp)
+ atomic_store_lse128_release
+ } else {
+ // if detect(FEAT_LSE2) => lse2 (stp)
+ atomic_store_lse2_release
+ }
} else {
- // if detect(FEAT_LSE2) => lse2 (stp)
- atomic_store_lse2_seqcst
+ // else => no_lse2:
+ atomic_store_no_lse2_release
}
- } else {
- // else => no_lse2:
- atomic_store_no_lse2_seqcst
- }
- });
+ });
+ }
+ Ordering::SeqCst => {
+ ifunc!(unsafe fn(dst: *mut u128, val: u128) {
+ let cpuinfo = detect::detect();
+ if cpuinfo.lse2() {
+ if cpuinfo.lse128() {
+ // if detect(FEAT_LSE2) && detect(FEAT_LSE128) && order == seqcst => lse128 (swpp)
+ atomic_store_lse128_seqcst
+ } else if cpuinfo.rcpc3() {
+ // if detect(FEAT_LSE2) && detect(FEAT_LRCPC3) && order != relaxed => lse2_rcpc3 (stilp)
+ atomic_store_lse2_rcpc3_seqcst
+ } else {
+ // if detect(FEAT_LSE2) => lse2 (stp)
+ atomic_store_lse2_seqcst
+ }
+ } else {
+ // else => no_lse2:
+ atomic_store_no_lse2_seqcst
+ }
+ });
+ }
+ _ => unreachable!(),
}
- _ => unreachable!(),
}
}
}
// else => no_lse2:
- #[cfg(not(all(
- not(portable_atomic_no_outline_atomics),
- any(
- all(
- target_os = "linux",
- any(
- target_env = "gnu",
- all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
- ),
- portable_atomic_outline_atomics,
- ),
- ),
- target_os = "android",
- target_os = "freebsd",
- target_os = "netbsd",
- target_os = "openbsd",
- all(target_os = "illumos", portable_atomic_outline_atomics),
- // These don't support detection of FEAT_LSE2.
- // target_os = "fuchsia",
- // target_os = "windows",
- ),
- )))]
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- atomic_store_no_lse2(dst, val, order);
+ #[cfg(else)]
+ {
+ use self::atomic_store_no_lse2 as atomic_store;
}
-}
+});
+#[cfg(not(any(target_feature = "lse2", portable_atomic_target_feature = "lse2")))]
+cfg_sel!({
+ // if compile_time(FEAT_LSE) && not(ll_sc_rmw) => casp
+ // If FEAT_LSE is available at compile-time and portable_atomic_ll_sc_rmw cfg is not set,
+ // we use CAS-based atomic RMW.
+ #[cfg(all(
+ any(target_feature = "lse", portable_atomic_target_feature = "lse"),
+ not(portable_atomic_ll_sc_rmw),
+ ))]
+ {
+ #[inline]
+ unsafe fn atomic_store_no_lse2(dst: *mut u128, val: u128, order: Ordering) {
+ // SAFETY: the caller must uphold the safety contract.
+ // cfg guarantee that the CPU supports FEAT_LSE.
+ unsafe {
+ _atomic_swap_casp(dst, val, order);
+ }
+ }
+ }
+ // else => ldxp_stxp
+ #[cfg(else)]
+ {
+ use self::_atomic_store_ldxp_stxp as atomic_store_no_lse2;
+ }
+});
// If CPU supports FEAT_LSE2, STP/STILP is single-copy atomic writes,
// otherwise it is two single-copy atomic writes.
// Refs: B2.2.1 of the Arm Architecture Reference Manual Armv8, for Armv8-A architecture profile
@@ -1077,9 +1046,8 @@ unsafe fn _atomic_store_stp(dst: *mut u128, val: u128, order: Ordering) {
// SAFETY: the caller must guarantee that `dst` is valid for writes,
// 16-byte aligned, that there are no concurrent non-atomic operations.
//
- // Refs: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/STP--Store-pair-of-registers-
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/STP--Store-pair-of-registers-
unsafe {
- #[rustfmt::skip]
macro_rules! atomic_store {
($acquire:tt, $release:tt) => {{
let val = U128 { whole: val };
@@ -1154,7 +1122,7 @@ unsafe fn _atomic_store_stilp(dst: *mut u128, val: u128, order: Ordering) {
// SAFETY: the caller must guarantee that `dst` is valid for writes,
// 16-byte aligned, that there are no concurrent non-atomic operations.
//
- // Refs: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/STILP--Store-release-ordered-pair-of-registers-
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/STILP--Store-release-ordered-pair-of-registers-
unsafe {
macro_rules! atomic_store {
($acquire:tt) => {{
@@ -1173,7 +1141,7 @@ unsafe fn _atomic_store_stilp(dst: *mut u128, val: u128, order: Ordering) {
// https://github.com/llvm/llvm-project/commit/a6aaa969f7caec58a994142f8d855861cf3a1463
#[cfg(portable_atomic_pre_llvm_16)]
asm!(
- // 0: d9031802 stilp x2, x3, [x0]
+ // stilp x2, x3, [x0]
".inst 0xd9031802",
$acquire,
in("x0") ptr_reg!(dst),
@@ -1185,7 +1153,7 @@ unsafe fn _atomic_store_stilp(dst: *mut u128, val: u128, order: Ordering) {
}
match order {
Ordering::Release => atomic_store!(""),
- // LLVM uses store-release (dmb ish; stp); dmb ish, GCC (libatomic)
+ // LLVM uses store-release (dmb ish; stp); dmb ish, GCC (libatomic) and Atomics ABI Standard
// uses store-release (stilp) without fence for SeqCst store
// (https://github.com/gcc-mirror/gcc/commit/7107574958e2bed11d916a1480ef1319f15e5ffe).
// Considering https://reviews.llvm.org/D141748, LLVM's lowing seems
@@ -1266,9 +1234,11 @@ unsafe fn atomic_compare_exchange(
any(
target_env = "gnu",
all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
@@ -1278,7 +1248,7 @@ unsafe fn atomic_compare_exchange(
target_os = "openbsd",
all(target_os = "illumos", portable_atomic_outline_atomics),
target_os = "fuchsia",
- target_os = "windows",
+ windows,
),
))]
#[cfg(not(any(target_feature = "lse", portable_atomic_target_feature = "lse")))]
@@ -1324,7 +1294,7 @@ unsafe fn atomic_compare_exchange(
match success {
Ordering::Relaxed => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> u128 {
- if detect::detect().has_lse() {
+ if detect::detect().lse() {
// if detect(FEAT_LSE) => casp
atomic_compare_exchange_casp_relaxed
} else {
@@ -1335,7 +1305,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::Acquire => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> u128 {
- if detect::detect().has_lse() {
+ if detect::detect().lse() {
// if detect(FEAT_LSE) => casp
atomic_compare_exchange_casp_acquire
} else {
@@ -1346,7 +1316,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::Release => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> u128 {
- if detect::detect().has_lse() {
+ if detect::detect().lse() {
// if detect(FEAT_LSE) => casp
atomic_compare_exchange_casp_release
} else {
@@ -1359,7 +1329,7 @@ unsafe fn atomic_compare_exchange(
#[cfg(not(target_env = "msvc"))]
Ordering::AcqRel | Ordering::SeqCst => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> u128 {
- if detect::detect().has_lse() {
+ if detect::detect().lse() {
// if detect(FEAT_LSE) => casp
atomic_compare_exchange_casp_acqrel
} else {
@@ -1371,7 +1341,7 @@ unsafe fn atomic_compare_exchange(
#[cfg(target_env = "msvc")]
Ordering::AcqRel => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> u128 {
- if detect::detect().has_lse() {
+ if detect::detect().lse() {
// if detect(FEAT_LSE) => casp
atomic_compare_exchange_casp_acqrel
} else {
@@ -1383,7 +1353,7 @@ unsafe fn atomic_compare_exchange(
#[cfg(target_env = "msvc")]
Ordering::SeqCst => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> u128 {
- if detect::detect().has_lse() {
+ if detect::detect().lse() {
// if detect(FEAT_LSE) => casp
atomic_compare_exchange_casp_seqcst
} else {
@@ -1405,9 +1375,11 @@ unsafe fn atomic_compare_exchange(
any(
target_env = "gnu",
all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
@@ -1417,17 +1389,13 @@ unsafe fn atomic_compare_exchange(
target_os = "openbsd",
all(target_os = "illumos", portable_atomic_outline_atomics),
target_os = "fuchsia",
- target_os = "windows",
+ windows,
),
)))]
#[cfg(not(any(target_feature = "lse", portable_atomic_target_feature = "lse")))]
// SAFETY: the caller must uphold the safety contract.
let prev = unsafe { _atomic_compare_exchange_ldxp_stxp(dst, old, new, success, failure) };
- if prev == old {
- Ok(prev)
- } else {
- Err(prev)
- }
+ if prev == old { Ok(prev) } else { Err(prev) }
}
#[cfg(any(
target_feature = "lse",
@@ -1450,7 +1418,7 @@ unsafe fn _atomic_compare_exchange_casp(
// reads, 16-byte aligned, that there are no concurrent non-atomic operations,
// and the CPU supports FEAT_LSE.
//
- // Refs: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/CASP--CASPA--CASPAL--CASPL--Compare-and-swap-pair-of-words-or-doublewords-in-memory-
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/CASP--CASPA--CASPAL--CASPL--Compare-and-swap-pair-of-words-or-doublewords-in-memory-
unsafe {
let old = U128 { whole: old };
let new = U128 { whole: new };
@@ -1492,10 +1460,10 @@ unsafe fn _atomic_compare_exchange_ldxp_stxp(
// reads, 16-byte aligned, and that there are no concurrent non-atomic operations.
//
// Refs:
- // - LDXP: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/LDXP--Load-exclusive-pair-of-registers-
- // - LDAXP: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/LDAXP--Load-acquire-exclusive-pair-of-registers-
- // - STXP: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/STXP--Store-exclusive-pair-of-registers-
- // - STLXP: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/STLXP--Store-release-exclusive-pair-of-registers-
+ // - LDXP: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/LDXP--Load-exclusive-pair-of-registers-
+ // - LDAXP: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/LDAXP--Load-acquire-exclusive-pair-of-registers-
+ // - STXP: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/STXP--Store-exclusive-pair-of-registers-
+ // - STLXP: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/STLXP--Store-release-exclusive-pair-of-registers-
//
// Note: Load-Exclusive pair (by itself) does not guarantee atomicity; to complete an atomic
// operation (even load/store), a corresponding Store-Exclusive pair must succeed.
@@ -1572,22 +1540,23 @@ Note:
*/
-// If FEAT_LSE is available at compile-time and portable_atomic_ll_sc_rmw cfg is not set,
-// we use CAS-based atomic RMW.
-#[cfg(not(any(target_feature = "lse128", portable_atomic_target_feature = "lse128")))]
-#[cfg(all(
- any(target_feature = "lse", portable_atomic_target_feature = "lse"),
- not(portable_atomic_ll_sc_rmw),
-))]
-use self::_atomic_swap_casp as atomic_swap;
-#[cfg(not(any(target_feature = "lse128", portable_atomic_target_feature = "lse128")))]
-#[cfg(not(all(
- any(target_feature = "lse", portable_atomic_target_feature = "lse"),
- not(portable_atomic_ll_sc_rmw),
-)))]
-use self::_atomic_swap_ldxp_stxp as atomic_swap;
-#[cfg(any(target_feature = "lse128", portable_atomic_target_feature = "lse128"))]
-use self::_atomic_swap_swpp as atomic_swap;
+cfg_sel!({
+ #[cfg(any(target_feature = "lse128", portable_atomic_target_feature = "lse128"))]
+ {
+ use self::_atomic_swap_swpp as atomic_swap;
+ }
+ #[cfg(all(
+ any(target_feature = "lse", portable_atomic_target_feature = "lse"),
+ not(portable_atomic_ll_sc_rmw),
+ ))]
+ {
+ use self::_atomic_swap_casp as atomic_swap;
+ }
+ #[cfg(else)]
+ {
+ use self::_atomic_swap_ldxp_stxp as atomic_swap;
+ }
+});
#[cfg(any(
target_feature = "lse128",
portable_atomic_target_feature = "lse128",
@@ -1605,7 +1574,7 @@ unsafe fn _atomic_swap_swpp(dst: *mut u128, val: u128, order: Ordering) -> u128
// reads, 16-byte aligned, that there are no concurrent non-atomic operations,
// and the CPU supports FEAT_LSE128.
//
- // Refs: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/SWPP--SWPPA--SWPPAL--SWPPL--Swap-quadword-in-memory-
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/SWPP--SWPPA--SWPPAL--SWPPL--Swap-quadword-in-memory-
unsafe {
let val = U128 { whole: val };
let (prev_lo, prev_hi);
@@ -1631,7 +1600,7 @@ unsafe fn _atomic_swap_swpp(dst: *mut u128, val: u128, order: Ordering) -> u128
macro_rules! swap {
($order:tt, $fence:tt) => {
asm!(
- // 4: 19{2,a,6,e}18002 swpp{,a,l,al} x2, x1, [x0]
+ // swpp{,a,l,al} x2, x1, [x0]
concat!(".inst 0x19", $order, "18002"),
$fence,
in("x0") ptr_reg!(dst),
@@ -2026,7 +1995,7 @@ unsafe fn atomic_and(dst: *mut u128, val: u128, order: Ordering) -> u128 {
// reads, 16-byte aligned, that there are no concurrent non-atomic operations,
// and the CPU supports FEAT_LSE128.
//
- // Refs: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/LDCLRP--LDCLRPA--LDCLRPAL--LDCLRPL--Atomic-bit-clear-on-quadword-in-memory-
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/LDCLRP--LDCLRPA--LDCLRPAL--LDCLRPL--Atomic-bit-clear-on-quadword-in-memory-
unsafe {
let val = U128 { whole: !val };
let (prev_lo, prev_hi);
@@ -2052,7 +2021,7 @@ unsafe fn atomic_and(dst: *mut u128, val: u128, order: Ordering) -> u128 {
macro_rules! clear {
($order:tt, $fence:tt) => {
asm!(
- // 8: 19{2,a,6,e}11008 ldclrp{,a,l,al} x8, x1, [x0]
+ // ldclrp{,a,l,al} x8, x1, [x0]
concat!(".inst 0x19", $order, "11008"),
$fence,
in("x0") ptr_reg!(dst),
@@ -2104,7 +2073,7 @@ unsafe fn atomic_or(dst: *mut u128, val: u128, order: Ordering) -> u128 {
// reads, 16-byte aligned, that there are no concurrent non-atomic operations,
// and the CPU supports FEAT_LSE128.
//
- // Refs: https://developer.arm.com/documentation/ddi0602/2024-06/Base-Instructions/LDSETP--LDSETPA--LDSETPAL--LDSETPL--Atomic-bit-set-on-quadword-in-memory-
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/Base-Instructions/LDSETP--LDSETPA--LDSETPAL--LDSETPL--Atomic-bit-set-on-quadword-in-memory-
unsafe {
let val = U128 { whole: val };
let (prev_lo, prev_hi);
@@ -2130,7 +2099,7 @@ unsafe fn atomic_or(dst: *mut u128, val: u128, order: Ordering) -> u128 {
macro_rules! or {
($order:tt, $fence:tt) => {
asm!(
- // 4: 19{2,a,6,e}13002 ldsetp{,a,l,al} x2, x1, [x0]
+ // ldsetp{,a,l,al} x2, x1, [x0]
concat!(".inst 0x19", $order, "13002"),
$fence,
in("x0") ptr_reg!(dst),
### external/vendor/portable-atomic/src/imp/atomic128/intrinsics.rs
@@ -3,7 +3,7 @@
/*
128-bit atomic implementation without inline assembly.
-Adapted from https://github.com/rust-lang/rust/blob/1.80.0/library/core/src/sync/atomic.rs.
+Adapted from https://github.com/rust-lang/rust/blob/1.84.0/library/core/src/sync/atomic.rs.
Note: This module is currently only enabled on Miri and ThreadSanitizer which
do not support inline assembly.
@@ -25,6 +25,9 @@ Note:
- On powerpc64, LLVM (as of 17) doesn't support 128-bit atomic min/max:
https://github.com/llvm/llvm-project/issues/68390
- On powerpc64le, LLVM (as of 17) generates broken code. (wrong result from fetch_add)
+- On riscv64, LLVM does not automatically use 128-bit atomic instructions even if zacas feature is
+ enabled, because doing it changes the ABI. (If the ability to do that is provided by LLVM in the
+ future, it should probably be controlled by another ABI feature similar to forced-atomics.)
*/
include!("macros.rs");
@@ -40,21 +43,17 @@ mod fallback;
#[path = "../detect/x86_64.rs"]
mod detect;
-use core::sync::atomic::Ordering;
#[cfg(not(target_arch = "x86_64"))]
-use core::{
- intrinsics,
- sync::atomic::Ordering::{AcqRel, Acquire, Relaxed, Release, SeqCst},
-};
+use core::intrinsics;
+use core::sync::atomic::Ordering::{self, AcqRel, Acquire, Relaxed, Release, SeqCst};
-// https://github.com/rust-lang/rust/blob/1.80.0/library/core/src/sync/atomic.rs#L3267
#[cfg(target_arch = "x86_64")]
#[inline]
fn strongest_failure_ordering(order: Ordering) -> Ordering {
match order {
- Ordering::Release | Ordering::Relaxed => Ordering::Relaxed,
- Ordering::SeqCst => Ordering::SeqCst,
- Ordering::Acquire | Ordering::AcqRel => Ordering::Acquire,
+ Release | Relaxed => Relaxed,
+ SeqCst => SeqCst,
+ Acquire | AcqRel => Acquire,
_ => unreachable!(),
}
}
@@ -126,7 +125,7 @@ unsafe fn atomic_compare_exchange(
debug_assert!(dst as usize % 16 == 0);
#[cfg(not(target_feature = "cmpxchg16b"))]
{
- debug_assert!(detect::detect().has_cmpxchg16b());
+ debug_assert!(detect::detect().cmpxchg16b());
}
// SAFETY: the caller must guarantee that `dst` is valid for both writes and
// reads, 16-byte aligned (required by CMPXCHG16B), that there are no
@@ -148,7 +147,7 @@ unsafe fn atomic_compare_exchange(
ifunc!(unsafe fn(
dst: *mut u128, old: u128, new: u128, success: Ordering, failure: Ordering
) -> (u128, bool) {
- if detect::detect().has_cmpxchg16b() {
+ if detect::detect().cmpxchg16b() {
cmpxchg16b
} else {
fallback::atomic_compare_exchange
@@ -178,11 +177,7 @@ unsafe fn atomic_compare_exchange(
_ => unreachable!(),
}
};
- if ok {
- Ok(val)
- } else {
- Err(val)
- }
+ if ok { Ok(val) } else { Err(val) }
}
#[cfg(target_arch = "x86_64")]
@@ -218,11 +213,7 @@ unsafe fn atomic_compare_exchange_weak(
_ => unreachable!(),
}
};
- if ok {
- Ok(val)
- } else {
- Err(val)
- }
+ if ok { Ok(val) } else { Err(val) }
}
#[inline(always)]
@@ -493,7 +484,7 @@ fn is_lock_free() -> bool {
}
#[cfg(not(target_feature = "cmpxchg16b"))]
{
- detect::detect().has_cmpxchg16b()
+ detect::detect().cmpxchg16b()
}
}
#[cfg(target_arch = "x86_64")]
### external/vendor/portable-atomic/src/imp/atomic128/mod.rs
@@ -8,20 +8,35 @@ See README.md for details.
// AArch64
#[cfg(any(
- all(target_arch = "aarch64", any(not(portable_atomic_no_asm), portable_atomic_unstable_asm)),
- all(target_arch = "arm64ec", not(portable_atomic_no_asm))
+ all(
+ target_arch = "aarch64",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ ),
+ all(
+ target_arch = "arm64ec",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
+ not(portable_atomic_no_asm),
+ ),
))]
// Use intrinsics.rs on Miri and Sanitizer that do not support inline assembly.
-#[cfg_attr(
- all(any(miri, portable_atomic_sanitize_thread), portable_atomic_new_atomic_intrinsics),
- path = "intrinsics.rs"
-)]
+#[cfg_attr(any(miri, portable_atomic_sanitize_thread), path = "intrinsics.rs")]
pub(super) mod aarch64;
// powerpc64
#[cfg(all(
target_arch = "powerpc64",
- portable_atomic_unstable_asm_experimental_arch,
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ not(portable_atomic_atomic_intrinsics),
+ )),
+ not(portable_atomic_no_asm),
any(
target_feature = "quadword-atomics",
portable_atomic_target_feature = "quadword-atomics",
@@ -37,49 +52,61 @@ pub(super) mod aarch64;
any(target_endian = "little", not(target_feature = "crt-static")),
),
all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
target_os = "android",
- target_os = "freebsd",
+ all(
+ target_os = "freebsd",
+ any(
+ target_endian = "little",
+ not(target_feature = "crt-static"),
+ portable_atomic_outline_atomics,
+ ),
+ ),
target_os = "openbsd",
+ all(
+ target_os = "aix",
+ not(portable_atomic_pre_llvm_20),
+ any(test, portable_atomic_outline_atomics), // TODO(aix): currently disabled by default
+ ),
),
not(any(miri, portable_atomic_sanitize_thread)),
),
),
))]
// Use intrinsics.rs on Miri and Sanitizer that do not support inline assembly.
-#[cfg_attr(
- all(any(miri, portable_atomic_sanitize_thread), not(portable_atomic_pre_llvm_15)),
- path = "intrinsics.rs"
-)]
+#[cfg_attr(any(miri, portable_atomic_sanitize_thread), path = "intrinsics.rs")]
pub(super) mod powerpc64;
// riscv64
#[cfg(all(
target_arch = "riscv64",
- not(portable_atomic_no_asm),
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
+ target_feature = "zacas",
+ portable_atomic_target_feature = "zacas",
all(
feature = "fallback",
not(portable_atomic_no_outline_atomics),
- any(test, portable_atomic_outline_atomics), // TODO(riscv): currently disabled by default
any(target_os = "linux", target_os = "android"),
- not(any(miri, portable_atomic_sanitize_thread)),
),
),
))]
-// Use intrinsics.rs on Miri and Sanitizer that do not support inline assembly.
-#[cfg_attr(any(miri, portable_atomic_sanitize_thread), path = "intrinsics.rs")]
pub(super) mod riscv64;
// s390x
-#[cfg(all(target_arch = "s390x", not(portable_atomic_no_asm)))]
+#[cfg(all(
+ target_arch = "s390x",
+ not(all(any(miri, portable_atomic_sanitize_thread), not(portable_atomic_atomic_intrinsics))),
+ not(portable_atomic_no_asm),
+))]
// Use intrinsics.rs on Miri and Sanitizer that do not support inline assembly.
#[cfg_attr(any(miri, portable_atomic_sanitize_thread), path = "intrinsics.rs")]
pub(super) mod s390x;
### external/vendor/portable-atomic/src/imp/atomic128/powerpc64.rs
@@ -29,9 +29,7 @@ Refs:
- atomic-maybe-uninit
https://github.com/taiki-e/atomic-maybe-uninit
-Generated asm:
-- powerpc64 (pwr8) https://godbolt.org/z/TjKsPbWc6
-- powerpc64le https://godbolt.org/z/5WqPGhb3Y
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
include!("macros.rs");
@@ -61,19 +59,38 @@ mod fallback;
target_env = "gnu",
any(target_endian = "little", not(target_feature = "crt-static")),
),
- all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
- ),
+ all(target_env = "musl", any(not(target_feature = "crt-static"), feature = "std")),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
portable_atomic_outline_atomics,
),
),
target_os = "android",
- target_os = "freebsd",
+ all(
+ target_os = "freebsd",
+ any(
+ target_endian = "little",
+ not(target_feature = "crt-static"),
+ portable_atomic_outline_atomics,
+ ),
+ ),
target_os = "openbsd",
))]
#[path = "../detect/auxv.rs"]
mod detect;
+#[cfg(not(portable_atomic_no_outline_atomics))]
+#[cfg(any(
+ test,
+ not(any(
+ target_feature = "quadword-atomics",
+ portable_atomic_target_feature = "quadword-atomics",
+ )),
+))]
+#[cfg(target_os = "aix")]
+#[cfg(not(portable_atomic_pre_llvm_20))] // SIGTRAP on LLVM 19
+#[cfg(any(test, portable_atomic_outline_atomics))] // TODO(aix): currently disabled by default
+#[path = "../detect/powerpc64_aix.rs"]
+mod detect;
use core::{arch::asm, sync::atomic::Ordering};
@@ -86,7 +103,7 @@ macro_rules! debug_assert_pwr8 {
portable_atomic_target_feature = "quadword-atomics",
)))]
{
- debug_assert!(detect::detect().has_quadword_atomics());
+ debug_assert!(detect::detect().quadword_atomics());
}
};
}
@@ -190,7 +207,7 @@ unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
match order {
Ordering::Relaxed => {
ifunc!(unsafe fn(src: *mut u128) -> u128 {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
atomic_load_pwr8_relaxed
} else {
fallback::atomic_load_non_seqcst
@@ -199,7 +216,7 @@ unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
}
Ordering::Acquire => {
ifunc!(unsafe fn(src: *mut u128) -> u128 {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
atomic_load_pwr8_acquire
} else {
fallback::atomic_load_non_seqcst
@@ -208,7 +225,7 @@ unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
}
Ordering::SeqCst => {
ifunc!(unsafe fn(src: *mut u128) -> u128 {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
atomic_load_pwr8_seqcst
} else {
fallback::atomic_load_seqcst
@@ -300,7 +317,7 @@ unsafe fn atomic_store(dst: *mut u128, val: u128, order: Ordering) {
match order {
Ordering::Relaxed => {
ifunc!(unsafe fn(dst: *mut u128, val: u128) {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
atomic_store_pwr8_relaxed
} else {
fallback::atomic_store_non_seqcst
@@ -309,7 +326,7 @@ unsafe fn atomic_store(dst: *mut u128, val: u128, order: Ordering) {
}
Ordering::Release => {
ifunc!(unsafe fn(dst: *mut u128, val: u128) {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
atomic_store_pwr8_release
} else {
fallback::atomic_store_non_seqcst
@@ -318,7 +335,7 @@ unsafe fn atomic_store(dst: *mut u128, val: u128, order: Ordering) {
}
Ordering::SeqCst => {
ifunc!(unsafe fn(dst: *mut u128, val: u128) {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
atomic_store_pwr8_seqcst
} else {
fallback::atomic_store_seqcst
@@ -403,7 +420,7 @@ unsafe fn atomic_compare_exchange(
match success {
Ordering::Relaxed => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_relaxed_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -412,7 +429,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::Acquire => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_acquire_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -421,7 +438,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::Release => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_release_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -430,7 +447,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::AcqRel => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_acqrel_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -439,7 +456,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::SeqCst => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_seqcst_fn
} else {
fallback::atomic_compare_exchange_seqcst
@@ -450,11 +467,7 @@ unsafe fn atomic_compare_exchange(
}
}
};
- if ok {
- Ok(prev)
- } else {
- Err(prev)
- }
+ if ok { Ok(prev) } else { Err(prev) }
}
#[inline]
unsafe fn atomic_compare_exchange_pwr8(
@@ -536,11 +549,7 @@ unsafe fn atomic_compare_exchange_weak(
// SAFETY: the caller must uphold the safety contract.
// cfg guarantees that quadword atomics instructions are available at compile-time.
let (prev, ok) = unsafe { atomic_compare_exchange_weak_pwr8(dst, old, new, success, failure) };
- if ok {
- Ok(prev)
- } else {
- Err(prev)
- }
+ if ok { Ok(prev) } else { Err(prev) }
}
#[cfg(any(
target_feature = "quadword-atomics",
@@ -866,7 +875,7 @@ macro_rules! select_atomic_rmw {
match order {
Ordering::Relaxed => {
ifunc!(unsafe fn($($arg)*) $(-> $ret_ty)? {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_relaxed_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -875,7 +884,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::Acquire => {
ifunc!(unsafe fn($($arg)*) $(-> $ret_ty)? {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_acquire_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -884,7 +893,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::Release => {
ifunc!(unsafe fn($($arg)*) $(-> $ret_ty)? {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_release_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -893,7 +902,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::AcqRel => {
ifunc!(unsafe fn($($arg)*) $(-> $ret_ty)? {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_acqrel_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -902,7 +911,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::SeqCst => {
ifunc!(unsafe fn($($arg)*) $(-> $ret_ty)? {
- if detect::detect().has_quadword_atomics() {
+ if detect::detect().quadword_atomics() {
pwr8_seqcst_fn
} else {
fallback::$seqcst_fallback_fn
@@ -1010,7 +1019,7 @@ fn is_lock_free() -> bool {
portable_atomic_target_feature = "quadword-atomics",
)))]
{
- detect::detect().has_quadword_atomics()
+ detect::detect().quadword_atomics()
}
}
const IS_ALWAYS_LOCK_FREE: bool = cfg!(any(
@@ -1021,6 +1030,7 @@ const IS_ALWAYS_LOCK_FREE: bool = cfg!(any(
atomic128!(AtomicI128, i128, atomic_max, atomic_min);
atomic128!(AtomicU128, u128, atomic_umax, atomic_umin);
+#[cfg(not(valgrind))] // TODO(powerpc64): Hang (as of Valgrind 3.26)
#[cfg(test)]
mod tests {
use super::*;
### external/vendor/portable-atomic/src/imp/atomic128/riscv64.rs
@@ -17,86 +17,75 @@ this module and use fallback implementation instead.
Refs:
- RISC-V Instruction Set Manual
"Zacas" Extension for Atomic Compare-and-Swap (CAS) Instructions
- https://github.com/riscv/riscv-isa-manual/blob/riscv-isa-release-8b9dc50-2024-08-30/src/zacas.adoc
+ https://github.com/riscv/riscv-isa-manual/blob/riscv-isa-release-56e76be-2025-08-26/src/zacas.adoc
- RISC-V Atomics ABI Specification
- https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/draft-20240829-13bfa9f54634cb60d86b9b333e109f077805b4b3/riscv-atomic.adoc
+ https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/draft-20250812-301374e92976e298e676e7129a6212926b2299ce/riscv-atomic.adoc
-Generated asm:
-- riscv64gc (+experimental-zacas) https://godbolt.org/z/hPY86hWcc
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
include!("macros.rs");
-#[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
-)))]
+#[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
#[path = "../fallback/outline_atomics.rs"]
mod fallback;
#[cfg(not(portable_atomic_no_outline_atomics))]
-#[cfg(any(test, portable_atomic_outline_atomics))] // TODO(riscv): currently disabled by default
-#[cfg(any(
- test,
- not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )),
-))]
+#[cfg(any(test, not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))))]
#[cfg(any(target_os = "linux", target_os = "android"))]
#[path = "../detect/riscv_linux.rs"]
mod detect;
-use core::{arch::asm, sync::atomic::Ordering};
+#[cfg(not(portable_atomic_no_asm))]
+use core::arch::asm;
+use core::sync::atomic::Ordering;
use crate::utils::{Pair, U128};
macro_rules! debug_assert_zacas {
() => {
- #[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )))]
+ #[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
{
- debug_assert!(detect::detect().has_zacas());
+ debug_assert!(detect::detect().zacas());
}
};
}
-// LLVM doesn't support `.option arch, +zabha` directive as of LLVM 19 because it is experimental.
-// So, we currently always using .4byte directive.
+// `.option arch, +zacas` directive requires LLVM 20, so we use .4byte directive for old LLVM.
// Note that `.insn <value>` directive requires LLVM 19.
// https://github.com/llvm/llvm-project/commit/2a086dce691e3cc34a2fc27f4fb255bb2cbbfac9
-// // https://github.com/riscv-non-isa/riscv-asm-manual/blob/ad0de8c004e29c9a7ac33cfd054f4d4f9392f2fb/src/asm-manual.adoc#arch
-// macro_rules! start_zacas {
-// () => {
-// ".option push\n.option arch, +zacas"
-// };
-// }
-// macro_rules! end_zacas {
-// () => {
-// ".option pop"
-// };
-// }
+// https://github.com/riscv-non-isa/riscv-asm-manual/blob/v0.0.1/src/asm-manual.adoc#arch
+#[cfg(not(portable_atomic_pre_llvm_20))]
+macro_rules! start_zacas {
+ () => {
+ ".option push\n.option arch, +zacas"
+ };
+}
+#[cfg(not(portable_atomic_pre_llvm_20))]
+macro_rules! end_zacas {
+ () => {
+ ".option pop"
+ };
+}
-// LLVM doesn't support `.option arch, +zabha` directive as of LLVM 19 because it is experimental.
-// So, we currently always using .4byte directive.
-// macro_rules! atomic_rmw_amocas_order {
-// ($op:ident, $order:ident) => {
-// atomic_rmw_amocas_order!($op, $order, failure = $order)
-// };
-// ($op:ident, $order:ident, failure = $failure:ident) => {
-// match $order {
-// Ordering::Relaxed => $op!("", ""),
-// Ordering::Acquire => $op!("", ".aq"),
-// Ordering::Release => $op!("", ".rl"),
-// Ordering::AcqRel => $op!("", ".aqrl"),
-// Ordering::SeqCst if $failure == Ordering::SeqCst => $op!("fence rw,rw", ".aqrl"),
-// Ordering::SeqCst => $op!("", ".aqrl"),
-// _ => unreachable!(),
-// }
-// };
-// }
+#[cfg(not(portable_atomic_pre_llvm_20))]
+macro_rules! atomic_rmw_amocas_order {
+ ($op:ident, $order:ident) => {
+ atomic_rmw_amocas_order!($op, $order, failure = $order)
+ };
+ ($op:ident, $order:ident, failure = $failure:ident) => {
+ match $order {
+ Ordering::Relaxed => $op!("", ""),
+ Ordering::Acquire => $op!("", ".aq"),
+ Ordering::Release => $op!("", ".rl"),
+ Ordering::AcqRel => $op!("", ".aqrl"),
+ Ordering::SeqCst if $failure == Ordering::SeqCst => $op!("fence rw,rw", ".aqrl"),
+ Ordering::SeqCst => $op!("", ".aqrl"),
+ _ => unreachable!(),
+ }
+ };
+}
+#[cfg(portable_atomic_pre_llvm_20)]
macro_rules! atomic_rmw_amocas_order_insn {
($op:ident, $order:ident) => {
atomic_rmw_amocas_order_insn!($op, $order, failure = $order)
@@ -115,16 +104,10 @@ macro_rules! atomic_rmw_amocas_order_insn {
}
// If zacas is available at compile-time, we can always use zacas_fn.
-#[cfg(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
-))]
+#[cfg(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))]
use self::atomic_load_zacas as atomic_load;
// Otherwise, we need to do run-time detection and can use zacas_fn only if zacas is available.
-#[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
-)))]
+#[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
#[inline]
unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
fn_alias! {
@@ -142,7 +125,7 @@ unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
match order {
Ordering::Relaxed => {
ifunc!(unsafe fn(src: *mut u128) -> u128 {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
atomic_load_zacas_relaxed
} else {
fallback::atomic_load_non_seqcst
@@ -151,7 +134,7 @@ unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
}
Ordering::Acquire => {
ifunc!(unsafe fn(src: *mut u128) -> u128 {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
atomic_load_zacas_acquire
} else {
fallback::atomic_load_non_seqcst
@@ -160,7 +143,7 @@ unsafe fn atomic_load(src: *mut u128, order: Ordering) -> u128 {
}
Ordering::SeqCst => {
ifunc!(unsafe fn(src: *mut u128) -> u128 {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
atomic_load_zacas_seqcst
} else {
fallback::atomic_load_seqcst
@@ -179,23 +162,24 @@ unsafe fn atomic_load_zacas(src: *mut u128, order: Ordering) -> u128 {
// SAFETY: the caller must uphold the safety contract.
unsafe {
- // LLVM doesn't support `.option arch, +zabha` directive as of LLVM 19 because it is experimental.
- // So, we currently always using .4byte directive.
- // macro_rules! load {
- // ($fence:tt, $asm_order:tt) => {
- // asm!(
- // start_zacas!(),
- // $fence, // fence
- // concat!("amocas.q", $asm_order, " a2, a2, 0({src})"), // atomic { if *dst == a2:a3 { *dst = a2:a3 } else { a2:a3 = *dst } }
- // end_zacas!(),
- // src = in(reg) ptr_reg!(src),
- // inout("a2") 0_u64 => out_lo,
- // inout("a3") 0_u64 => out_hi,
- // options(nostack, preserves_flags),
- // )
- // };
- // }
- // atomic_rmw_amocas_order!(load, order);
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ macro_rules! load {
+ ($fence:tt, $asm_order:tt) => {
+ asm!(
+ start_zacas!(),
+ $fence, // fence
+ concat!("amocas.q", $asm_order, " a2, a2, 0({src})"), // atomic { if *dst == a2:a3 { *dst = a2:a3 } else { a2:a3 = *dst } }
+ end_zacas!(),
+ src = in(reg) ptr_reg!(src),
+ inout("a2") 0_u64 => out_lo,
+ inout("a3") 0_u64 => out_hi,
+ options(nostack, preserves_flags),
+ )
+ };
+ }
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ atomic_rmw_amocas_order!(load, order);
+ #[cfg(portable_atomic_pre_llvm_20)]
macro_rules! load {
($fence:tt, $insn_order:tt) => {
asm!(
@@ -209,6 +193,7 @@ unsafe fn atomic_load_zacas(src: *mut u128, order: Ordering) -> u128 {
)
};
}
+ #[cfg(portable_atomic_pre_llvm_20)]
atomic_rmw_amocas_order_insn!(load, order);
U128 { pair: Pair { lo: out_lo, hi: out_hi } }.whole
}
@@ -230,17 +215,11 @@ unsafe fn atomic_compare_exchange(
success: Ordering,
failure: Ordering,
) -> Result<u128, u128> {
- #[cfg(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- ))]
+ #[cfg(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))]
// SAFETY: the caller must uphold the safety contract.
// cfg guarantees that zacas instructions are available at compile-time.
let (prev, ok) = unsafe { atomic_compare_exchange_zacas(dst, old, new, success, failure) };
- #[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )))]
+ #[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
let (prev, ok) = {
fn_alias! {
// inline(never) is just a hint and also not strictly necessary
@@ -260,7 +239,7 @@ unsafe fn atomic_compare_exchange(
match order {
Ordering::Relaxed => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_relaxed_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -269,7 +248,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::Acquire => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_acquire_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -278,7 +257,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::Release => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_release_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -287,7 +266,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::AcqRel => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_acqrel_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -296,7 +275,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::SeqCst => {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_seqcst_fn
} else {
fallback::atomic_compare_exchange_seqcst
@@ -307,11 +286,7 @@ unsafe fn atomic_compare_exchange(
}
}
};
- if ok {
- Ok(prev)
- } else {
- Err(prev)
- }
+ if ok { Ok(prev) } else { Err(prev) }
}
#[inline]
unsafe fn atomic_compare_exchange_zacas(
@@ -330,27 +305,28 @@ unsafe fn atomic_compare_exchange_zacas(
// SAFETY: the caller must uphold the safety contract.
unsafe {
- // LLVM doesn't support `.option arch, +zabha` directive as of LLVM 19 because it is experimental.
- // So, we currently always using .4byte directive.
- // macro_rules! cmpxchg {
- // ($fence:tt, $asm_order:tt) => {
- // asm!(
- // start_zacas!(),
- // $fence, // fence
- // concat!("amocas.q", $asm_order, " a4, a2, 0({dst})"), // atomic { if *dst == a4:a5 { *dst = a2:a3 } else { a4:a5 = *dst } }
- // end_zacas!(),
- // dst = in(reg) ptr_reg!(dst),
- // // must be allocated to even/odd register pair
- // inout("a4") old.pair.lo => prev_lo,
- // inout("a5") old.pair.hi => prev_hi,
- // // must be allocated to even/odd register pair
- // in("a2") new.pair.lo,
- // in("a3") new.pair.hi,
- // options(nostack, preserves_flags),
- // )
- // };
- // }
- // atomic_rmw_amocas_order!(cmpxchg, order, failure = failure);
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ macro_rules! cmpxchg {
+ ($fence:tt, $asm_order:tt) => {
+ asm!(
+ start_zacas!(),
+ $fence, // fence
+ concat!("amocas.q", $asm_order, " a4, a2, 0({dst})"), // atomic { if *dst == a4:a5 { *dst = a2:a3 } else { a4:a5 = *dst } }
+ end_zacas!(),
+ dst = in(reg) ptr_reg!(dst),
+ // must be allocated to even/odd register pair
+ inout("a4") old.pair.lo => prev_lo,
+ inout("a5") old.pair.hi => prev_hi,
+ // must be allocated to even/odd register pair
+ in("a2") new.pair.lo,
+ in("a3") new.pair.hi,
+ options(nostack, preserves_flags),
+ )
+ };
+ }
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ atomic_rmw_amocas_order!(cmpxchg, order, failure = failure);
+ #[cfg(portable_atomic_pre_llvm_20)]
macro_rules! cmpxchg {
($fence:tt, $insn_order:tt) => {
asm!(
@@ -368,6 +344,7 @@ unsafe fn atomic_compare_exchange_zacas(
)
};
}
+ #[cfg(portable_atomic_pre_llvm_20)]
atomic_rmw_amocas_order_insn!(cmpxchg, order, failure = failure);
let prev = U128 { pair: Pair { lo: prev_lo, hi: prev_hi } }.whole;
(prev, prev == old.whole)
@@ -386,7 +363,7 @@ unsafe fn byte_wise_atomic_load(src: *const u128) -> u128 {
unsafe {
asm!(
"ld {out_lo}, ({src})", // atomic { out_lo = *src }
- "ld {out_hi}, 8({src})", // atomic { out_hi = *src.add(8) }
+ "ld {out_hi}, 8({src})", // atomic { out_hi = *src.byte_add(8) }
src = in(reg) ptr_reg!(src),
out_lo = out(reg) out_lo,
out_hi = out(reg) out_hi,
@@ -430,16 +407,10 @@ macro_rules! select_atomic_rmw {
}
}
// If zacas is available at compile-time, we can always use zacas_fn.
- #[cfg(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- ))]
+ #[cfg(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))]
use self::$zacas_fn as $name;
// Otherwise, we need to do run-time detection and can use zacas_fn only if zacas is available.
- #[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )))]
+ #[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
#[inline]
unsafe fn $name(dst: *mut u128 $(, $($arg)*)?, order: Ordering) $(-> $ret_ty)? {
fn_alias! {
@@ -459,7 +430,7 @@ macro_rules! select_atomic_rmw {
match order {
Ordering::Relaxed => {
ifunc!(unsafe fn(dst: *mut u128 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_relaxed_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -468,7 +439,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::Acquire => {
ifunc!(unsafe fn(dst: *mut u128 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_acquire_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -477,7 +448,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::Release => {
ifunc!(unsafe fn(dst: *mut u128 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_release_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -486,7 +457,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::AcqRel => {
ifunc!(unsafe fn(dst: *mut u128 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_acqrel_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -495,7 +466,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::SeqCst => {
ifunc!(unsafe fn(dst: *mut u128 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_seqcst_fn
} else {
fallback::$seqcst_fallback_fn
@@ -622,31 +593,22 @@ select_atomic_rmw! {
#[inline]
fn is_lock_free() -> bool {
- #[cfg(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- ))]
+ #[cfg(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))]
{
// zacas is available at compile-time.
true
}
- #[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )))]
+ #[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
{
- detect::detect().has_zacas()
+ detect::detect().zacas()
}
}
-const IS_ALWAYS_LOCK_FREE: bool = cfg!(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
-));
+const IS_ALWAYS_LOCK_FREE: bool =
+ cfg!(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"));
atomic128!(AtomicI128, i128, atomic_max, atomic_min);
atomic128!(AtomicU128, u128, atomic_umax, atomic_umin);
-#[allow(clippy::undocumented_unsafe_blocks, clippy::wildcard_imports)]
#[cfg(test)]
mod tests {
use super::*;
### external/vendor/portable-atomic/src/imp/atomic128/s390x.rs
@@ -13,24 +13,20 @@ detailed description of the atomic and synchronize instructions in this architec
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#s390x
LLVM's minimal supported architecture level is arch8 (z10):
-https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/SystemZ/SystemZProcessors.td#L16-L17
+https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/SystemZ/SystemZProcessors.td#L16-L17
This does not appear to have changed since the current s390x backend was added in LLVM 3.3:
https://github.com/llvm/llvm-project/commit/5f613dfd1f7edb0ae95d521b7107b582d9df5103#diff-cbaef692b3958312e80fd5507a7e2aff071f1acb086f10e8a96bc06a7bb289db
Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use intrinsics.rs instead.
Refs:
-- z/Architecture Principles of Operation, Fourteenth Edition (SA22-7832-13)
- https://publibfp.dhe.ibm.com/epubs/pdf/a227832d.pdf
+- z/Architecture Principles of Operation, Fifteenth Edition (SA22-7832-14)
+ https://www.ibm.com/docs/en/module_1678991624569/pdf/SA22-7832-14.pdf
- atomic-maybe-uninit
https://github.com/taiki-e/atomic-maybe-uninit
-Generated asm:
-- s390x https://godbolt.org/z/oPxYYEvPG
-- s390x (z196) https://godbolt.org/z/M69KrKT7Y
-- s390x (z15,-vector) https://godbolt.org/z/Wec8b3ada
-- s390x (z15) https://godbolt.org/z/KxWcrbfYh
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
include!("macros.rs");
@@ -188,11 +184,7 @@ unsafe fn atomic_compare_exchange(
);
U128 { pair: Pair { hi: prev_hi, lo: prev_lo } }.whole
};
- if extract_cc(r) {
- Ok(prev)
- } else {
- Err(prev)
- }
+ if extract_cc(r) { Ok(prev) } else { Err(prev) }
}
// cdsg is always strong.
@@ -209,7 +201,7 @@ unsafe fn byte_wise_atomic_load(src: *const u128) -> u128 {
unsafe {
let (out_hi, out_lo);
asm!(
- "lg {out_hi}, 8({src})", // atomic { out_hi = *src.add(8) }
+ "lg {out_hi}, 8({src})", // atomic { out_hi = *src.byte_add(8) }
"lg {out_lo}, 0({src})", // atomic { out_lo = *src }
src = in(reg) src,
out_hi = out(reg) out_hi,
@@ -264,7 +256,7 @@ unsafe fn atomic_swap(dst: *mut u128, val: u128, _order: Ordering) -> u128 {
unsafe {
// atomic swap is always SeqCst.
asm!(
- "lg %r0, 8({dst})", // atomic { r0 = *dst.add(8) }
+ "lg %r0, 8({dst})", // atomic { r0 = *dst.byte_add(8) }
"lg %r1, 0({dst})", // atomic { r1 = *dst }
"2:", // 'retry:
"cdsg %r0, %r12, 0({dst})", // atomic { if *dst == r0:r1 { cc = 0; *dst = r12:r13 } else { cc = 1; r0:r1 = *dst } }
@@ -303,7 +295,7 @@ macro_rules! atomic_rmw_cas_3 {
unsafe {
// atomic RMW is always SeqCst.
asm!(
- "lg %r0, 8({dst})", // atomic { r0 = *dst.add(8) }
+ "lg %r0, 8({dst})", // atomic { r0 = *dst.byte_add(8) }
"lg %r1, 0({dst})", // atomic { r1 = *dst }
"2:", // 'retry:
$($op)*
@@ -345,7 +337,7 @@ macro_rules! atomic_rmw_cas_2 {
unsafe {
// atomic RMW is always SeqCst.
asm!(
- "lg %r0, 8({dst})", // atomic { r0 = *dst.add(8) }
+ "lg %r0, 8({dst})", // atomic { r0 = *dst.byte_add(8) }
"lg %r1, 0({dst})", // atomic { r1 = *dst }
"2:", // 'retry:
$($op)*
### external/vendor/portable-atomic/src/imp/atomic128/x86_64.rs
@@ -1,7 +1,12 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
/*
-128-bit atomic implementation on x86_64 using CMPXCHG16B (DWCAS).
+128-bit atomic implementation on x86_64.
+
+This architecture provides the following 128-bit atomic instructions:
+
+- CMPXCHG16B: CAS (CMPXCHG16B)
+- VMOVDQA: load/store (AVX)
Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use intrinsics.rs instead.
@@ -10,8 +15,7 @@ Refs:
- x86 and amd64 instruction reference https://www.felixcloutier.com/x86
- atomic-maybe-uninit https://github.com/taiki-e/atomic-maybe-uninit
-Generated asm:
-- x86_64 (+cmpxchg16b) https://godbolt.org/z/rfs1jxd51
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
// TODO: use core::arch::x86_64::cmpxchg16b where available and efficient than asm
@@ -28,6 +32,13 @@ mod fallback;
not(target_feature = "sse"),
cfg(not(any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b")))
)]
+#[cfg(any(
+ test,
+ not(all(
+ any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
+ target_feature = "avx",
+ )),
+))]
#[path = "../detect/x86_64.rs"]
mod detect;
@@ -45,29 +56,41 @@ macro_rules! debug_assert_cmpxchg16b {
portable_atomic_target_feature = "cmpxchg16b",
)))]
{
- debug_assert!(detect::detect().has_cmpxchg16b());
+ debug_assert!(detect::detect().cmpxchg16b());
}
};
}
-#[cfg(not(any(portable_atomic_no_outline_atomics, target_env = "sgx")))]
#[cfg(target_feature = "sse")]
-macro_rules! debug_assert_vmovdqa_atomic {
+#[cfg(not(all(
+ not(target_feature = "avx"),
+ any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
+)))]
+macro_rules! debug_assert_cmpxchg16b_avx {
() => {{
debug_assert_cmpxchg16b!();
- debug_assert!(detect::detect().has_vmovdqa_atomic());
+ #[cfg(not(target_feature = "avx"))]
+ {
+ debug_assert!(detect::detect().avx());
+ }
}};
}
-#[cfg(not(any(portable_atomic_no_outline_atomics, target_env = "sgx")))]
#[cfg(target_feature = "sse")]
+#[cfg(not(all(
+ not(target_feature = "avx"),
+ any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
+)))]
#[cfg(target_pointer_width = "32")]
macro_rules! ptr_modifier {
() => {
":e"
};
}
-#[cfg(not(any(portable_atomic_no_outline_atomics, target_env = "sgx")))]
#[cfg(target_feature = "sse")]
+#[cfg(not(all(
+ not(target_feature = "avx"),
+ any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
+)))]
#[cfg(target_pointer_width = "64")]
macro_rules! ptr_modifier {
() => {
@@ -110,49 +133,65 @@ unsafe fn cmpxchg16b(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
let new = U128 { whole: new };
let (prev_lo, prev_hi);
macro_rules! cmpxchg16b {
- ($rdi:tt) => {
+ ($dst:tt) => {
asm!(
- "xchg {rbx_tmp}, rbx", // save rbx which is reserved by LLVM
- concat!("lock cmpxchg16b xmmword ptr [", $rdi, "]"),
- "sete cl",
- "mov rbx, {rbx_tmp}", // restore rbx
- rbx_tmp = inout(reg) new.pair.lo => _,
+ "xchg r8, rbx", // save rbx which is reserved by LLVM
+ concat!("lock cmpxchg16b xmmword ptr [", $dst, "]"),
+ "setne cl",
+ "mov rbx, r8", // restore rbx
+ inout("r8") new.pair.lo => _,
in("rcx") new.pair.hi,
inout("rax") old.pair.lo => prev_lo,
inout("rdx") old.pair.hi => prev_hi,
- in($rdi) dst,
+ in($dst) dst,
lateout("cl") r,
// Do not use `preserves_flags` because CMPXCHG16B modifies the ZF flag.
options(nostack),
)
};
}
+ // rdi and rsi are call-preserved on Windows.
+ #[cfg(not(windows))]
+ #[cfg(target_pointer_width = "32")]
+ cmpxchg16b!("esi");
+ #[cfg(not(windows))]
+ #[cfg(target_pointer_width = "64")]
+ cmpxchg16b!("rsi");
+ #[cfg(windows)]
#[cfg(target_pointer_width = "32")]
- cmpxchg16b!("edi");
+ cmpxchg16b!("r11d");
+ #[cfg(windows)]
#[cfg(target_pointer_width = "64")]
- cmpxchg16b!("rdi");
+ cmpxchg16b!("r11");
crate::utils::assert_unchecked(r == 0 || r == 1); // needed to remove extra test
- (U128 { pair: Pair { lo: prev_lo, hi: prev_hi } }.whole, r != 0)
+ (U128 { pair: Pair { lo: prev_lo, hi: prev_hi } }.whole, r == 0)
}
}
-// VMOVDQA is atomic on Intel, AMD, and Zhaoxin CPUs with AVX.
+// VMOVDQA is atomic when AVX available.
// See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104688 for details.
+// Intel, AMD, and Zhaoxin officially guarantee this behavior (VIA has not responded),
+// but both LLVM and Boost always treat VMOVDQA as atomic when AVX available.
+// https://github.com/llvm/llvm-project/pull/74275
+// https://github.com/boostorg/atomic/commit/24a41db3e61627d99895f7e324b3d725d1be27c1
//
// Refs: https://www.felixcloutier.com/x86/movdqa:vmovdqa32:vmovdqa64
//
// Use cfg(target_feature = "sse") here -- SSE is included in the x86_64
// baseline and is always available, but the SSE target feature is disabled for
// use cases such as kernels and firmware that should not use vector registers.
// So, do not use vector registers unless SSE target feature is enabled.
-// See also https://github.com/rust-lang/rust/blob/1.80.0/src/doc/rustc/src/platform-support/x86_64-unknown-none.md.
-#[cfg(not(any(portable_atomic_no_outline_atomics, target_env = "sgx")))]
+// See also https://github.com/rust-lang/rust/blob/1.84.0/src/doc/rustc/src/platform-support/x86_64-unknown-none.md.
#[cfg(target_feature = "sse")]
+#[cfg(not(all(
+ not(target_feature = "avx"),
+ any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
+)))]
#[target_feature(enable = "avx")]
#[inline]
-unsafe fn atomic_load_vmovdqa(src: *mut u128) -> u128 {
+unsafe fn _atomic_load_vmovdqa(src: *mut u128) -> u128 {
debug_assert!(src as usize % 16 == 0);
- debug_assert_vmovdqa_atomic!();
+ debug_assert_cmpxchg16b_avx!();
// SAFETY: the caller must uphold the safety contract.
//
@@ -168,13 +207,16 @@ unsafe fn atomic_load_vmovdqa(src: *mut u128) -> u128 {
core::mem::transmute(out)
}
}
-#[cfg(not(any(portable_atomic_no_outline_atomics, target_env = "sgx")))]
#[cfg(target_feature = "sse")]
+#[cfg(not(all(
+ not(target_feature = "avx"),
+ any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
+)))]
#[target_feature(enable = "avx")]
#[inline]
-unsafe fn atomic_store_vmovdqa(dst: *mut u128, val: u128, order: Ordering) {
+unsafe fn _atomic_store_vmovdqa(dst: *mut u128, val: u128, order: Ordering) {
debug_assert!(dst as usize % 16 == 0);
- debug_assert_vmovdqa_atomic!();
+ debug_assert_cmpxchg16b_avx!();
// SAFETY: the caller must uphold the safety contract.
unsafe {
@@ -193,17 +235,17 @@ unsafe fn atomic_store_vmovdqa(dst: *mut u128, val: u128, order: Ordering) {
let p = core::cell::UnsafeCell::new(core::mem::MaybeUninit::<u64>::uninit());
asm!(
concat!("vmovdqa xmmword ptr [{dst", ptr_modifier!(), "}], {val}"),
- // Equivalent to mfence, but is up to 3.1x faster on Coffee Lake and up to 2.4x faster on Raptor Lake-H at least in simple cases.
+ // Equivalent to `mfence`, but is up to 3.1x faster on Coffee Lake and up to 2.4x faster on Raptor Lake-H at least in simple cases.
// - https://github.com/taiki-e/portable-atomic/pull/156
- // - LLVM uses lock or for x86_32 64-bit atomic SeqCst store using SSE https://godbolt.org/z/9sKEr8YWc
- // - Windows uses xchg for x86_32 for MemoryBarrier https://learn.microsoft.com/en-us/windows/win32/api/winnt/nf-winnt-memorybarrier
- // - MSVC STL uses lock inc https://github.com/microsoft/STL/pull/740
- // - boost uses lock or https://github.com/boostorg/atomic/commit/559eba81af71386cedd99f170dc6101c6ad7bf22
+ // - LLVM uses `lock or` https://godbolt.org/z/vv6rjzfYd
+ // - Windows uses `xchg` for x86_32 for MemoryBarrier https://learn.microsoft.com/en-us/windows/win32/api/winnt/nf-winnt-memorybarrier
+ // - MSVC STL uses `lock inc` https://github.com/microsoft/STL/pull/740
+ // - boost uses `lock or` https://github.com/boostorg/atomic/commit/559eba81af71386cedd99f170dc6101c6ad7bf22
concat!("xchg qword ptr [{p", ptr_modifier!(), "}], {tmp}"),
dst = in(reg) dst,
val = in(xmm_reg) val,
- p = inout(reg) p.get() => _,
- tmp = lateout(reg) _,
+ p = in(reg) p.get(),
+ tmp = out(reg) _,
options(nostack, preserves_flags),
);
}
@@ -212,6 +254,10 @@ unsafe fn atomic_store_vmovdqa(dst: *mut u128, val: u128, order: Ordering) {
}
}
+#[cfg(not(all(
+ any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
+ target_feature = "avx",
+)))]
#[cfg(not(all(
any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
@@ -229,63 +275,85 @@ macro_rules! load_store_detect {
)))]
{
// Check CMPXCHG16B first to prevent mixing atomic and non-atomic access.
- if cpuid.has_cmpxchg16b() {
- // We only use VMOVDQA when SSE is enabled. See atomic_load_vmovdqa() for more.
- #[cfg(target_feature = "sse")]
+ if cpuid.cmpxchg16b() {
+ #[cfg(target_feature = "avx")]
{
- if cpuid.has_vmovdqa_atomic() {
- $vmovdqa
- } else {
- $cmpxchg16b
- }
+ $vmovdqa
}
- #[cfg(not(target_feature = "sse"))]
+ // We only use VMOVDQA when SSE is enabled. See _atomic_load_vmovdqa() for more.
+ #[cfg(not(target_feature = "avx"))]
{
- $cmpxchg16b
+ #[cfg(target_feature = "sse")]
+ {
+ if cpuid.avx() { $vmovdqa } else { $cmpxchg16b }
+ }
+ #[cfg(not(target_feature = "sse"))]
+ {
+ $cmpxchg16b
+ }
}
} else {
fallback::$fallback
}
}
#[cfg(any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"))]
{
- if cpuid.has_vmovdqa_atomic() {
- $vmovdqa
- } else {
- $cmpxchg16b
- }
+ if cpuid.avx() { $vmovdqa } else { $cmpxchg16b }
}
}};
}
#[inline]
unsafe fn atomic_load(src: *mut u128, _order: Ordering) -> u128 {
- // We only use VMOVDQA when SSE is enabled. See atomic_load_vmovdqa() for more.
- // SGX doesn't support CPUID.
#[cfg(all(
any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
- any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
+ target_feature = "avx",
))]
// SAFETY: the caller must uphold the safety contract.
- // cfg guarantees that CMPXCHG16B is available at compile-time.
+ // cfg guarantees that CMPXCHG16B and AVX are available at compile-time.
unsafe {
- // cmpxchg16b is always SeqCst.
- atomic_load_cmpxchg16b(src)
+ _atomic_load_vmovdqa(src)
}
#[cfg(not(all(
any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
- any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
+ target_feature = "avx",
)))]
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- ifunc!(unsafe fn(src: *mut u128) -> u128 {
- load_store_detect! {
- vmovdqa = atomic_load_vmovdqa
- cmpxchg16b = atomic_load_cmpxchg16b
- // Use SeqCst because cmpxchg16b and atomic load by vmovdqa is always SeqCst.
- fallback = atomic_load_seqcst
- }
- })
+ {
+ // We only use VMOVDQA when SSE is enabled. See _atomic_load_vmovdqa() for more.
+ // SGX doesn't support CPUID.
+ #[cfg(all(
+ any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
+ any(
+ portable_atomic_no_outline_atomics,
+ target_env = "sgx",
+ not(target_feature = "sse")
+ ),
+ ))]
+ // SAFETY: the caller must uphold the safety contract.
+ // cfg guarantees that CMPXCHG16B is available at compile-time.
+ unsafe {
+ // cmpxchg16b is always SeqCst.
+ _atomic_load_cmpxchg16b(src)
+ }
+ #[cfg(not(all(
+ any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
+ any(
+ portable_atomic_no_outline_atomics,
+ target_env = "sgx",
+ not(target_feature = "sse")
+ ),
+ )))]
+ // SAFETY: the caller must uphold the safety contract.
+ unsafe {
+ ifunc!(unsafe fn(src: *mut u128) -> u128 {
+ load_store_detect! {
+ vmovdqa = _atomic_load_vmovdqa
+ cmpxchg16b = _atomic_load_cmpxchg16b
+ // Use SeqCst because cmpxchg16b and atomic load by vmovdqa is always SeqCst.
+ fallback = atomic_load_seqcst
+ }
+ })
+ }
}
}
// See cmpxchg16b() for target_feature(enable).
@@ -294,7 +362,7 @@ unsafe fn atomic_load(src: *mut u128, _order: Ordering) -> u128 {
target_feature(enable = "cmpxchg16b")
)]
#[inline]
-unsafe fn atomic_load_cmpxchg16b(src: *mut u128) -> u128 {
+unsafe fn _atomic_load_cmpxchg16b(src: *mut u128) -> u128 {
debug_assert!(src as usize % 16 == 0);
debug_assert_cmpxchg16b!();
@@ -310,83 +378,115 @@ unsafe fn atomic_load_cmpxchg16b(src: *mut u128) -> u128 {
// cmpxchg16b is always SeqCst.
let (out_lo, out_hi);
macro_rules! cmpxchg16b {
- ($rdi:tt) => {
+ ($dst:tt, $save:tt) => {
asm!(
- "mov {rbx_tmp}, rbx", // save rbx which is reserved by LLVM
+ concat!("mov ", $save, ", rbx"), // save rbx which is reserved by LLVM
"xor rbx, rbx", // zeroed rbx
- concat!("lock cmpxchg16b xmmword ptr [", $rdi, "]"),
- "mov rbx, {rbx_tmp}", // restore rbx
+ concat!("lock cmpxchg16b xmmword ptr [", $dst, "]"),
+ concat!("mov rbx, ", $save), // restore rbx
// set old/new args of cmpxchg16b to 0 (rbx is zeroed after saved to rbx_tmp, to avoid xchg)
- rbx_tmp = out(reg) _,
+ out($save) _,
in("rcx") 0_u64,
inout("rax") 0_u64 => out_lo,
inout("rdx") 0_u64 => out_hi,
- in($rdi) src,
+ in($dst) src,
// Do not use `preserves_flags` because CMPXCHG16B modifies the ZF flag.
options(nostack),
)
};
}
+ // rdi and rsi are call-preserved on Windows.
+ #[cfg(not(windows))]
+ #[cfg(target_pointer_width = "32")]
+ cmpxchg16b!("edi", "rsi");
+ #[cfg(not(windows))]
+ #[cfg(target_pointer_width = "64")]
+ cmpxchg16b!("rdi", "rsi");
+ #[cfg(windows)]
#[cfg(target_pointer_width = "32")]
- cmpxchg16b!("edi");
+ cmpxchg16b!("r9d", "r8");
+ #[cfg(windows)]
#[cfg(target_pointer_width = "64")]
- cmpxchg16b!("rdi");
+ cmpxchg16b!("r9", "r8");
U128 { pair: Pair { lo: out_lo, hi: out_hi } }.whole
}
}
#[inline]
unsafe fn atomic_store(dst: *mut u128, val: u128, order: Ordering) {
- // We only use VMOVDQA when SSE is enabled. See atomic_load_vmovdqa() for more.
- // SGX doesn't support CPUID.
#[cfg(all(
any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
- any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
+ target_feature = "avx",
))]
// SAFETY: the caller must uphold the safety contract.
- // cfg guarantees that CMPXCHG16B is available at compile-time.
+ // cfg guarantees that CMPXCHG16B and AVX are available at compile-time.
unsafe {
- // cmpxchg16b is always SeqCst.
- let _ = order;
- atomic_store_cmpxchg16b(dst, val);
+ _atomic_store_vmovdqa(dst, val, order);
}
#[cfg(not(all(
any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
- any(portable_atomic_no_outline_atomics, target_env = "sgx", not(target_feature = "sse")),
+ target_feature = "avx",
)))]
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- #[cfg(target_feature = "sse")]
- fn_alias! {
- #[target_feature(enable = "avx")]
- unsafe fn(dst: *mut u128, val: u128);
- // atomic store by vmovdqa has at least release semantics.
- atomic_store_vmovdqa_non_seqcst = atomic_store_vmovdqa(Ordering::Release);
- atomic_store_vmovdqa_seqcst = atomic_store_vmovdqa(Ordering::SeqCst);
+ {
+ // We only use VMOVDQA when SSE is enabled. See _atomic_load_vmovdqa() for more.
+ // SGX doesn't support CPUID.
+ #[cfg(all(
+ any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
+ any(
+ portable_atomic_no_outline_atomics,
+ target_env = "sgx",
+ not(target_feature = "sse")
+ ),
+ ))]
+ // SAFETY: the caller must uphold the safety contract.
+ // cfg guarantees that CMPXCHG16B is available at compile-time.
+ unsafe {
+ // cmpxchg16b is always SeqCst.
+ let _ = order;
+ _atomic_store_cmpxchg16b(dst, val);
}
- match order {
- // Relaxed and Release stores are equivalent in all implementations
- // that may be called here (vmovdqa, asm-based cmpxchg16b, and fallback).
- // core::arch's cmpxchg16b will never called here.
- Ordering::Relaxed | Ordering::Release => {
- ifunc!(unsafe fn(dst: *mut u128, val: u128) {
- load_store_detect! {
- vmovdqa = atomic_store_vmovdqa_non_seqcst
- cmpxchg16b = atomic_store_cmpxchg16b
- fallback = atomic_store_non_seqcst
- }
- });
+ #[cfg(not(all(
+ any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b"),
+ any(
+ portable_atomic_no_outline_atomics,
+ target_env = "sgx",
+ not(target_feature = "sse")
+ ),
+ )))]
+ // SAFETY: the caller must uphold the safety contract.
+ unsafe {
+ #[cfg(target_feature = "sse")]
+ fn_alias! {
+ #[target_feature(enable = "avx")]
+ unsafe fn(dst: *mut u128, val: u128);
+ // atomic store by vmovdqa has at least release semantics.
+ _atomic_store_vmovdqa_non_seqcst = _atomic_store_vmovdqa(Ordering::Release);
+ _atomic_store_vmovdqa_seqcst = _atomic_store_vmovdqa(Ordering::SeqCst);
}
- Ordering::SeqCst => {
- ifunc!(unsafe fn(dst: *mut u128, val: u128) {
- load_store_detect! {
- vmovdqa = atomic_store_vmovdqa_seqcst
- cmpxchg16b = atomic_store_cmpxchg16b
- fallback = atomic_store_seqcst
- }
- });
+ match order {
+ // Relaxed and Release stores are equivalent in all implementations
+ // that may be called here (vmovdqa, asm-based cmpxchg16b, and fallback).
+ // core::arch's cmpxchg16b will never called here.
+ Ordering::Relaxed | Ordering::Release => {
+ ifunc!(unsafe fn(dst: *mut u128, val: u128) {
+ load_store_detect! {
+ vmovdqa = _atomic_store_vmovdqa_non_seqcst
+ cmpxchg16b = _atomic_store_cmpxchg16b
+ fallback = atomic_store_non_seqcst
+ }
+ });
+ }
+ Ordering::SeqCst => {
+ ifunc!(unsafe fn(dst: *mut u128, val: u128) {
+ load_store_detect! {
+ vmovdqa = _atomic_store_vmovdqa_seqcst
+ cmpxchg16b = _atomic_store_cmpxchg16b
+ fallback = atomic_store_seqcst
+ }
+ });
+ }
+ _ => unreachable!(),
}
- _ => unreachable!(),
}
}
}
@@ -396,7 +496,7 @@ unsafe fn atomic_store(dst: *mut u128, val: u128, order: Ordering) {
target_feature(enable = "cmpxchg16b")
)]
#[inline]
-unsafe fn atomic_store_cmpxchg16b(dst: *mut u128, val: u128) {
+unsafe fn _atomic_store_cmpxchg16b(dst: *mut u128, val: u128) {
// SAFETY: the caller must uphold the safety contract.
unsafe {
// cmpxchg16b is always SeqCst.
@@ -422,19 +522,15 @@ unsafe fn atomic_compare_exchange(
// reads, 16-byte aligned, and that there are no different kinds of concurrent accesses.
let (prev, ok) = unsafe {
ifunc!(unsafe fn(dst: *mut u128, old: u128, new: u128) -> (u128, bool) {
- if detect::detect().has_cmpxchg16b() {
+ if detect::detect().cmpxchg16b() {
cmpxchg16b
} else {
// Use SeqCst because cmpxchg16b is always SeqCst.
fallback::atomic_compare_exchange_seqcst
}
})
};
- if ok {
- Ok(prev)
- } else {
- Err(prev)
- }
+ if ok { Ok(prev) } else { Err(prev) }
}
// cmpxchg16b is always strong.
@@ -465,9 +561,9 @@ unsafe fn atomic_swap_cmpxchg16b(dst: *mut u128, val: u128, _order: Ordering) ->
let val = U128 { whole: val };
let (mut prev_lo, mut prev_hi);
macro_rules! cmpxchg16b {
- ($rdi:tt) => {
+ ($dst:tt, $save:tt) => {
asm!(
- "xchg {rbx_tmp}, rbx", // save rbx which is reserved by LLVM
+ concat!("xchg ", $save, ", rbx"), // save rbx which is reserved by LLVM
// This is not single-copy atomic reads, but this is ok because subsequent
// CAS will check for consistency.
//
@@ -477,26 +573,35 @@ unsafe fn atomic_swap_cmpxchg16b(dst: *mut u128, val: u128, _order: Ordering) ->
// so we must use inline assembly to implement this.
// (i.e., byte-wise atomic based on the standard library's atomic types
// cannot be used here).
- concat!("mov rax, qword ptr [", $rdi, "]"),
- concat!("mov rdx, qword ptr [", $rdi, " + 8]"),
+ concat!("mov rax, qword ptr [", $dst, "]"),
+ concat!("mov rdx, qword ptr [", $dst, " + 8]"),
"2:",
- concat!("lock cmpxchg16b xmmword ptr [", $rdi, "]"),
+ concat!("lock cmpxchg16b xmmword ptr [", $dst, "]"),
"jne 2b",
- "mov rbx, {rbx_tmp}", // restore rbx
- rbx_tmp = inout(reg) val.pair.lo => _,
+ concat!("mov rbx, ", $save), // restore rbx
+ inout($save) val.pair.lo => _,
in("rcx") val.pair.hi,
out("rax") prev_lo,
out("rdx") prev_hi,
- in($rdi) dst,
+ in($dst) dst,
// Do not use `preserves_flags` because CMPXCHG16B modifies the ZF flag.
options(nostack),
)
};
}
+ // rdi and rsi are call-preserved on Windows.
+ #[cfg(not(windows))]
+ #[cfg(target_pointer_width = "32")]
+ cmpxchg16b!("edi", "rsi");
+ #[cfg(not(windows))]
+ #[cfg(target_pointer_width = "64")]
+ cmpxchg16b!("rdi", "rsi");
+ #[cfg(windows)]
#[cfg(target_pointer_width = "32")]
- cmpxchg16b!("edi");
+ cmpxchg16b!("r9d", "r8");
+ #[cfg(windows)]
#[cfg(target_pointer_width = "64")]
- cmpxchg16b!("rdi");
+ cmpxchg16b!("r9", "r8");
U128 { pair: Pair { lo: prev_lo, hi: prev_hi } }.whole
}
}
@@ -505,7 +610,7 @@ unsafe fn atomic_swap_cmpxchg16b(dst: *mut u128, val: u128, _order: Ordering) ->
/// `unsafe fn(dst: *mut u128, val: u128, order: Ordering) -> u128;`
///
/// `$op` can use the following registers:
-/// - rsi/r8 pair: val argument (read-only for `$op`)
+/// - r8/r9 pair: val argument (read-only for `$op`)
/// - rax/rdx pair: previous value loaded (read-only for `$op`)
/// - rbx/rcx pair: new value that will be stored
// We could use CAS loop by atomic_compare_exchange here, but using an inline assembly allows
@@ -531,9 +636,9 @@ macro_rules! atomic_rmw_cas_3 {
let val = U128 { whole: val };
let (mut prev_lo, mut prev_hi);
macro_rules! cmpxchg16b {
- ($rdi:tt) => {
+ ($dst:tt, $save:tt) => {
asm!(
- "mov {rbx_tmp}, rbx", // save rbx which is reserved by LLVM
+ concat!("mov ", $save, ", rbx"), // save rbx which is reserved by LLVM
// This is not single-copy atomic reads, but this is ok because subsequent
// CAS will check for consistency.
//
@@ -543,29 +648,38 @@ macro_rules! atomic_rmw_cas_3 {
// so we must use inline assembly to implement this.
// (i.e., byte-wise atomic based on the standard library's atomic types
// cannot be used here).
- concat!("mov rax, qword ptr [", $rdi, "]"),
- concat!("mov rdx, qword ptr [", $rdi, " + 8]"),
+ concat!("mov rax, qword ptr [", $dst, "]"),
+ concat!("mov rdx, qword ptr [", $dst, " + 8]"),
"2:",
$($op)*
- concat!("lock cmpxchg16b xmmword ptr [", $rdi, "]"),
+ concat!("lock cmpxchg16b xmmword ptr [", $dst, "]"),
"jne 2b",
- "mov rbx, {rbx_tmp}", // restore rbx
- rbx_tmp = out(reg) _,
+ concat!("mov rbx, ", $save), // restore rbx
+ out($save) _,
out("rcx") _,
out("rax") prev_lo,
out("rdx") prev_hi,
- in($rdi) dst,
- in("rsi") val.pair.lo,
- in("r8") val.pair.hi,
+ in($dst) dst,
+ in("r8") val.pair.lo,
+ in("r9") val.pair.hi,
// Do not use `preserves_flags` because CMPXCHG16B modifies the ZF flag.
options(nostack),
)
};
}
+ // rdi and rsi are call-preserved on Windows.
+ #[cfg(not(windows))]
#[cfg(target_pointer_width = "32")]
- cmpxchg16b!("edi");
+ cmpxchg16b!("edi", "r10");
+ #[cfg(not(windows))]
#[cfg(target_pointer_width = "64")]
- cmpxchg16b!("rdi");
+ cmpxchg16b!("rdi", "r10");
+ #[cfg(windows)]
+ #[cfg(target_pointer_width = "32")]
+ cmpxchg16b!("r10d", "r11");
+ #[cfg(windows)]
+ #[cfg(target_pointer_width = "64")]
+ cmpxchg16b!("r10", "r11");
U128 { pair: Pair { lo: prev_lo, hi: prev_hi } }.whole
}
}
@@ -599,9 +713,9 @@ macro_rules! atomic_rmw_cas_2 {
// cmpxchg16b is always SeqCst.
let (mut prev_lo, mut prev_hi);
macro_rules! cmpxchg16b {
- ($rdi:tt) => {
+ ($dst:tt, $save:tt) => {
asm!(
- "mov {rbx_tmp}, rbx", // save rbx which is reserved by LLVM
+ concat!("mov ", $save, ", rbx"), // save rbx which is reserved by LLVM
// This is not single-copy atomic reads, but this is ok because subsequent
// CAS will check for consistency.
//
@@ -611,27 +725,36 @@ macro_rules! atomic_rmw_cas_2 {
// so we must use inline assembly to implement this.
// (i.e., byte-wise atomic based on the standard library's atomic types
// cannot be used here).
- concat!("mov rax, qword ptr [", $rdi, "]"),
- concat!("mov rdx, qword ptr [", $rdi, " + 8]"),
+ concat!("mov rax, qword ptr [", $dst, "]"),
+ concat!("mov rdx, qword ptr [", $dst, " + 8]"),
"2:",
$($op)*
- concat!("lock cmpxchg16b xmmword ptr [", $rdi, "]"),
+ concat!("lock cmpxchg16b xmmword ptr [", $dst, "]"),
"jne 2b",
- "mov rbx, {rbx_tmp}", // restore rbx
- rbx_tmp = out(reg) _,
+ concat!("mov rbx, ", $save), // restore rbx
+ out($save) _,
out("rcx") _,
out("rax") prev_lo,
out("rdx") prev_hi,
- in($rdi) dst,
+ in($dst) dst,
// Do not use `preserves_flags` because CMPXCHG16B modifies the ZF flag.
options(nostack),
)
};
}
+ // rdi and rsi are call-preserved on Windows.
+ #[cfg(not(windows))]
+ #[cfg(target_pointer_width = "32")]
+ cmpxchg16b!("edi", "rsi");
+ #[cfg(not(windows))]
+ #[cfg(target_pointer_width = "64")]
+ cmpxchg16b!("rdi", "rsi");
+ #[cfg(windows)]
#[cfg(target_pointer_width = "32")]
- cmpxchg16b!("edi");
+ cmpxchg16b!("r9d", "r8");
+ #[cfg(windows)]
#[cfg(target_pointer_width = "64")]
- cmpxchg16b!("rdi");
+ cmpxchg16b!("r9", "r8");
U128 { pair: Pair { lo: prev_lo, hi: prev_hi } }.whole
}
}
@@ -641,46 +764,46 @@ macro_rules! atomic_rmw_cas_2 {
atomic_rmw_cas_3! {
atomic_add_cmpxchg16b,
"mov rbx, rax",
- "add rbx, rsi",
+ "add rbx, r8",
"mov rcx, rdx",
- "adc rcx, r8",
+ "adc rcx, r9",
}
atomic_rmw_cas_3! {
atomic_sub_cmpxchg16b,
"mov rbx, rax",
- "sub rbx, rsi",
+ "sub rbx, r8",
"mov rcx, rdx",
- "sbb rcx, r8",
+ "sbb rcx, r9",
}
atomic_rmw_cas_3! {
atomic_and_cmpxchg16b,
"mov rbx, rax",
- "and rbx, rsi",
+ "and rbx, r8",
"mov rcx, rdx",
- "and rcx, r8",
+ "and rcx, r9",
}
atomic_rmw_cas_3! {
atomic_nand_cmpxchg16b,
"mov rbx, rax",
- "and rbx, rsi",
+ "and rbx, r8",
"not rbx",
"mov rcx, rdx",
- "and rcx, r8",
+ "and rcx, r9",
"not rcx",
}
atomic_rmw_cas_3! {
atomic_or_cmpxchg16b,
"mov rbx, rax",
- "or rbx, rsi",
+ "or rbx, r8",
"mov rcx, rdx",
- "or rcx, r8",
+ "or rcx, r9",
}
atomic_rmw_cas_3! {
atomic_xor_cmpxchg16b,
"mov rbx, rax",
- "xor rbx, rsi",
+ "xor rbx, r8",
"mov rcx, rdx",
- "xor rcx, r8",
+ "xor rcx, r9",
}
atomic_rmw_cas_2! {
@@ -700,42 +823,42 @@ atomic_rmw_cas_2! {
atomic_rmw_cas_3! {
atomic_max_cmpxchg16b,
- "cmp rsi, rax",
- "mov rcx, r8",
+ "cmp r8, rax",
+ "mov rcx, r9",
"sbb rcx, rdx",
- "mov rcx, r8",
+ "mov rcx, r9",
"cmovl rcx, rdx",
- "mov rbx, rsi",
+ "mov rbx, r8",
"cmovl rbx, rax",
}
atomic_rmw_cas_3! {
atomic_umax_cmpxchg16b,
- "cmp rsi, rax",
- "mov rcx, r8",
+ "cmp r8, rax",
+ "mov rcx, r9",
"sbb rcx, rdx",
- "mov rcx, r8",
+ "mov rcx, r9",
"cmovb rcx, rdx",
- "mov rbx, rsi",
+ "mov rbx, r8",
"cmovb rbx, rax",
}
atomic_rmw_cas_3! {
atomic_min_cmpxchg16b,
- "cmp rsi, rax",
- "mov rcx, r8",
+ "cmp r8, rax",
+ "mov rcx, r9",
"sbb rcx, rdx",
- "mov rcx, r8",
+ "mov rcx, r9",
"cmovge rcx, rdx",
- "mov rbx, rsi",
+ "mov rbx, r8",
"cmovge rbx, rax",
}
atomic_rmw_cas_3! {
atomic_umin_cmpxchg16b,
- "cmp rsi, rax",
- "mov rcx, r8",
+ "cmp r8, rax",
+ "mov rcx, r9",
"sbb rcx, rdx",
- "mov rcx, r8",
+ "mov rcx, r9",
"cmovae rcx, rdx",
- "mov rbx, rsi",
+ "mov rbx, r8",
"cmovae rbx, rax",
}
@@ -769,7 +892,7 @@ macro_rules! select_atomic_rmw {
// we only calls cmpxchg16b_fn if cmpxchg16b is available.
unsafe {
ifunc!(unsafe fn($($arg)*) $(-> $ret_ty)? {
- if detect::detect().has_cmpxchg16b() {
+ if detect::detect().cmpxchg16b() {
cmpxchg16b_seqcst_fn
} else {
// Use SeqCst because cmpxchg16b is always SeqCst.
@@ -856,7 +979,7 @@ fn is_lock_free() -> bool {
}
#[cfg(not(any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b")))]
{
- detect::detect().has_cmpxchg16b()
+ detect::detect().cmpxchg16b()
}
}
const IS_ALWAYS_LOCK_FREE: bool =
@@ -865,7 +988,6 @@ const IS_ALWAYS_LOCK_FREE: bool =
atomic128!(AtomicI128, i128, atomic_max, atomic_min);
atomic128!(AtomicU128, u128, atomic_umax, atomic_umin);
-#[allow(clippy::undocumented_unsafe_blocks, clippy::wildcard_imports)]
#[cfg(test)]
mod tests {
use super::*;
### external/vendor/portable-atomic/src/imp/atomic64/README.md
@@ -10,7 +10,11 @@ Here is the table of targets that support 64-bit atomics and the instructions us
| ----------- | ---- | ----- | --- | --- | ---- |
| x86 | cmpxchg8b or fild or movlps or movq | cmpxchg8b or fistp or movlps | cmpxchg8b | cmpxchg8b | provided by `core::sync::atomic` |
| arm | ldrexd | ldrexd/strexd | ldrexd/strexd | ldrexd/strexd | provided by `core::sync::atomic` for Armv6+, otherwise provided by us for Linux/Android using kuser_cmpxchg64 (see [arm_linux.rs](arm_linux.rs) for more) |
-| riscv32 | amocas.d | amocas.d | amocas.d | amocas.d | Experimental because LLVM marking the corresponding target feature as experimental. Requires `experimental-zacas` target feature. Both compile-time and run-time detection are supported (run-time detection is currently disabled by default). <br> Requires rustc 1.59+ |
+| riscv32 | amocas.d | amocas.d | amocas.d | amocas.d | Requires `zacas` target feature. Both compile-time and run-time detection are supported. <br> Requires Rust 1.59+ |
+| hexagon | memd | memd | memd_locked | memd_locked | Unimplemented |
+| sparc | ldx | stx | casx | casx | Unimplemented (unsupported in LLVM). Requires `v8plus` and `v9` target feature (Linux is v8plus+v9 by default) |
+| m68k | cas2 | cas2 | cas2 | cas2 | Unimplemented (unsupported in LLVM). Requires M68020 or later (Linux is M68020 by default) |
+| mips32r6 | llwp | llwp/scwp | llwp/scwp | llwp/scwp | Unimplemented (unsupported in LLVM). Requires Release 6 Paired LL/SC family of instructions |
If `core::sync::atomic` provides 64-bit atomics, we use them.
On compiler versions or platforms where these are not supported, the fallback implementation is used.
### external/vendor/portable-atomic/src/imp/atomic64/arm_linux.rs
@@ -3,30 +3,56 @@
/*
64-bit atomic implementation using kuser_cmpxchg64 on pre-v6 Arm Linux/Android.
+See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
+detailed description of the atomic and synchronize instructions in this architecture:
+https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#arm
+
Refs:
-- https://github.com/torvalds/linux/blob/v6.11/Documentation/arch/arm/kernel_user_helpers.rst
+- https://github.com/torvalds/linux/blob/v6.16/Documentation/arch/arm/kernel_user_helpers.rst
- https://github.com/rust-lang/compiler-builtins/blob/compiler_builtins-v0.1.124/src/arm_linux.rs
+Note: __kuser_cmpxchg64 is always SeqCst.
+https://github.com/torvalds/linux/blob/v6.16/arch/arm/kernel/entry-armv.S#L700-L707
+
Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use fallback implementation instead.
*/
// TODO: Since Rust 1.64, the Linux kernel requirement for Rust when using std is 3.2+, so it should
// be possible to omit the dynamic kernel version check if the std feature is enabled on Rust 1.64+.
-// https://blog.rust-lang.org/2022/08/01/Increasing-glibc-kernel-requirements.html
+// https://blog.rust-lang.org/2022/08/01/Increasing-glibc-kernel-requirements
include!("macros.rs");
#[path = "../fallback/outline_atomics.rs"]
mod fallback;
+#[cfg(test)] // test-only (unused)
+#[cfg(not(portable_atomic_no_outline_atomics))]
+#[cfg(any(
+ all(
+ target_os = "linux",
+ any(
+ target_env = "gnu",
+ target_env = "musl",
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
+ ),
+ ),
+ target_os = "android",
+ target_os = "freebsd",
+ target_os = "openbsd",
+))]
+#[path = "../detect/auxv.rs"]
+mod test_detect_auxv;
+
#[cfg(not(portable_atomic_no_asm))]
use core::arch::asm;
use core::{mem, sync::atomic::Ordering};
use crate::utils::{Pair, U64};
-// https://github.com/torvalds/linux/blob/v6.11/Documentation/arch/arm/kernel_user_helpers.rst
+// https://github.com/torvalds/linux/blob/v6.16/Documentation/arch/arm/kernel_user_helpers.rst
const KUSER_HELPER_VERSION: usize = 0xFFFF0FFC;
// __kuser_helper_version >= 5 (kernel version 3.1+)
const KUSER_CMPXCHG64: usize = 0xFFFF0F60;
@@ -42,16 +68,15 @@ fn __kuser_helper_version() -> i32 {
// SAFETY: core assumes that at least __kuser_memory_barrier (__kuser_helper_version >= 3,
// kernel version 2.6.15+) is available on this platform. __kuser_helper_version
// is always available on such a platform.
- v = unsafe { (KUSER_HELPER_VERSION as *const i32).read() };
+ v = unsafe { crate::utils::ptr::with_exposed_provenance::<i32>(KUSER_HELPER_VERSION).read() };
CACHE.store(v, Ordering::Relaxed);
v
}
#[inline]
fn has_kuser_cmpxchg64() -> bool {
- // Note: detect_false cfg is intended to make it easy for portable-atomic developers to
- // test cases such as has_cmpxchg16b == false, has_lse == false,
- // __kuser_helper_version < 5, etc., and is not a public API.
- if cfg!(portable_atomic_test_outline_atomics_detect_false) {
+ // Note: detect_false cfg is intended to make it easy for developers to test
+ // cases where features usually available is not available, and is not a public API.
+ if cfg!(portable_atomic_test_detect_false) {
return false;
}
__kuser_helper_version() >= 5
@@ -61,7 +86,7 @@ unsafe fn __kuser_cmpxchg64(old_val: *const u64, new_val: *const u64, ptr: *mut
// SAFETY: the caller must uphold the safety contract.
unsafe {
let f: extern "C" fn(*const u64, *const u64, *mut u64) -> u32 =
- mem::transmute(KUSER_CMPXCHG64 as *const ());
+ mem::transmute(crate::utils::ptr::with_exposed_provenance::<()>(KUSER_CMPXCHG64));
f(old_val, new_val, ptr) == 0
}
}
@@ -73,8 +98,8 @@ unsafe fn byte_wise_atomic_load(src: *const u64) -> u64 {
unsafe {
let (out_lo, out_hi);
asm!(
- "ldr {out_lo}, [{src}]",
- "ldr {out_hi}, [{src}, #4]",
+ "ldr {out_lo}, [{src}]", // atomic { out_lo = *src }
+ "ldr {out_hi}, [{src}, #4]", // atomic { out_hi = *src.byte_add(4) }
src = in(reg) src,
out_lo = out(reg) out_lo,
out_hi = out(reg) out_hi,
@@ -84,54 +109,50 @@ unsafe fn byte_wise_atomic_load(src: *const u64) -> u64 {
}
}
-#[inline(always)]
-unsafe fn atomic_update_kuser_cmpxchg64<F>(dst: *mut u64, mut f: F) -> u64
-where
- F: FnMut(u64) -> u64,
-{
- debug_assert!(dst as usize % 8 == 0);
- debug_assert!(has_kuser_cmpxchg64());
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- loop {
- // This is not single-copy atomic reads, but this is ok because subsequent
- // CAS will check for consistency.
- //
- // Arm's memory model allow mixed-sized atomic access.
- // https://github.com/rust-lang/unsafe-code-guidelines/issues/345#issuecomment-1172891466
- //
- // Note that the C++20 memory model does not allow mixed-sized atomic access,
- // so we must use inline assembly to implement byte_wise_atomic_load.
- // (i.e., byte-wise atomic based on the standard library's atomic types
- // cannot be used here).
- let prev = byte_wise_atomic_load(dst);
- let next = f(prev);
- if __kuser_cmpxchg64(&prev, &next, dst) {
- return prev;
- }
- }
- }
-}
-
-macro_rules! atomic_with_ifunc {
+macro_rules! select_atomic {
(
- unsafe fn $name:ident($($arg:tt)*) $(-> $ret_ty:ty)? { $($kuser_cmpxchg64_fn_body:tt)* }
+ unsafe fn $name:ident($dst:ident: *mut u64 $(, $($arg:tt)*)?) $(-> $ret_ty:ty)? {
+ |$kuser_cmpxchg64_fn_binding:ident| $($kuser_cmpxchg64_fn_body:tt)*
+ }
fallback = $seqcst_fallback_fn:ident
) => {
#[inline]
- unsafe fn $name($($arg)*, _: Ordering) $(-> $ret_ty)? {
- unsafe fn kuser_cmpxchg64_fn($($arg)*) $(-> $ret_ty)? {
- $($kuser_cmpxchg64_fn_body)*
+ unsafe fn $name($dst: *mut u64 $(, $($arg)*)?, _: Ordering) $(-> $ret_ty)? {
+ unsafe fn kuser_cmpxchg64_fn($dst: *mut u64 $(, $($arg)*)?) $(-> $ret_ty)? {
+ debug_assert!($dst as usize % 8 == 0);
+ debug_assert!(has_kuser_cmpxchg64());
+ // SAFETY: the caller must uphold the safety contract.
+ unsafe {
+ loop {
+ // This is not single-copy atomic reads, but this is ok because subsequent
+ // CAS will check for consistency.
+ //
+ // Arm's memory model allow mixed-sized atomic access.
+ // https://github.com/rust-lang/unsafe-code-guidelines/issues/345#issuecomment-1172891466
+ //
+ // Note that the C++20 memory model does not allow mixed-sized atomic access,
+ // so we must use inline assembly to implement byte_wise_atomic_load.
+ // (i.e., byte-wise atomic based on the standard library's atomic types
+ // cannot be used here).
+ let prev = byte_wise_atomic_load($dst);
+ let next = {
+ let $kuser_cmpxchg64_fn_binding = prev;
+ $($kuser_cmpxchg64_fn_body)*
+ };
+ if __kuser_cmpxchg64(&prev, &next, $dst) {
+ return prev;
+ }
+ }
+ }
}
// SAFETY: the caller must uphold the safety contract.
// we only calls __kuser_cmpxchg64 if it is available.
unsafe {
- ifunc!(unsafe fn($($arg)*) $(-> $ret_ty)? {
+ ifunc!(unsafe fn($dst: *mut u64 $(, $($arg)*)?) $(-> $ret_ty)? {
if has_kuser_cmpxchg64() {
kuser_cmpxchg64_fn
} else {
// Use SeqCst because __kuser_cmpxchg64 is always SeqCst.
- // https://github.com/torvalds/linux/blob/v6.11/arch/arm/kernel/entry-armv.S#L692-L699
fallback::$seqcst_fallback_fn
}
})
@@ -140,24 +161,22 @@ macro_rules! atomic_with_ifunc {
};
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_load(src: *mut u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(src, |old| old) }
+ |old| old
}
fallback = atomic_load_seqcst
}
-atomic_with_ifunc! {
- unsafe fn atomic_store(dst: *mut u64, val: u64) {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |_| val); }
+#[inline]
+unsafe fn atomic_store(dst: *mut u64, val: u64, order: Ordering) {
+ // SAFETY: the caller must uphold the safety contract.
+ unsafe {
+ atomic_swap(dst, val, order);
}
- fallback = atomic_store_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_swap(dst: *mut u64, val: u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |_| val) }
+ |_x| val
}
fallback = atomic_swap_seqcst
}
@@ -170,10 +189,19 @@ unsafe fn atomic_compare_exchange(
_: Ordering,
) -> Result<u64, u64> {
unsafe fn kuser_cmpxchg64_fn(dst: *mut u64, old: u64, new: u64) -> (u64, bool) {
+ debug_assert!(dst as usize % 8 == 0);
+ debug_assert!(has_kuser_cmpxchg64());
// SAFETY: the caller must uphold the safety contract.
- let prev =
- unsafe { atomic_update_kuser_cmpxchg64(dst, |v| if v == old { new } else { v }) };
- (prev, prev == old)
+ unsafe {
+ loop {
+ // See select_atomic! for more.
+ let prev = byte_wise_atomic_load(dst);
+ let next = if prev == old { new } else { prev };
+ if __kuser_cmpxchg64(&prev, &next, dst) {
+ return (prev, prev == old);
+ }
+ }
+ }
}
// SAFETY: the caller must uphold the safety contract.
// we only calls __kuser_cmpxchg64 if it is available.
@@ -183,105 +211,88 @@ unsafe fn atomic_compare_exchange(
kuser_cmpxchg64_fn
} else {
// Use SeqCst because __kuser_cmpxchg64 is always SeqCst.
- // https://github.com/torvalds/linux/blob/v6.11/arch/arm/kernel/entry-armv.S#L692-L699
fallback::atomic_compare_exchange_seqcst
}
})
};
- if ok {
- Ok(prev)
- } else {
- Err(prev)
- }
+ if ok { Ok(prev) } else { Err(prev) }
}
use self::atomic_compare_exchange as atomic_compare_exchange_weak;
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_add(dst: *mut u64, val: u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |x| x.wrapping_add(val)) }
+ |x| x.wrapping_add(val)
}
fallback = atomic_add_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_sub(dst: *mut u64, val: u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |x| x.wrapping_sub(val)) }
+ |x| x.wrapping_sub(val)
}
fallback = atomic_sub_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_and(dst: *mut u64, val: u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |x| x & val) }
+ |x| x & val
}
fallback = atomic_and_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_nand(dst: *mut u64, val: u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |x| !(x & val)) }
+ |x| !(x & val)
}
fallback = atomic_nand_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_or(dst: *mut u64, val: u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |x| x | val) }
+ |x| x | val
}
fallback = atomic_or_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_xor(dst: *mut u64, val: u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |x| x ^ val) }
+ |x| x ^ val
}
fallback = atomic_xor_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_max(dst: *mut u64, val: u64) -> u64 {
- #[allow(clippy::cast_possible_wrap, clippy::cast_sign_loss)]
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- atomic_update_kuser_cmpxchg64(dst, |x| core::cmp::max(x as i64, val as i64) as u64)
+ |x| {
+ #[allow(clippy::cast_possible_wrap, clippy::cast_sign_loss)]
+ { core::cmp::max(x as i64, val as i64) as u64 }
}
}
fallback = atomic_max_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_umax(dst: *mut u64, val: u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |x| core::cmp::max(x, val)) }
+ |x| core::cmp::max(x, val)
}
fallback = atomic_umax_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_min(dst: *mut u64, val: u64) -> u64 {
- #[allow(clippy::cast_possible_wrap, clippy::cast_sign_loss)]
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- atomic_update_kuser_cmpxchg64(dst, |x| core::cmp::min(x as i64, val as i64) as u64)
+ |x| {
+ #[allow(clippy::cast_possible_wrap, clippy::cast_sign_loss)]
+ { core::cmp::min(x as i64, val as i64) as u64 }
}
}
fallback = atomic_min_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_umin(dst: *mut u64, val: u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |x| core::cmp::min(x, val)) }
+ |x| core::cmp::min(x, val)
}
fallback = atomic_umin_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_not(dst: *mut u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, |x| !x) }
+ |x| !x
}
fallback = atomic_not_seqcst
}
-atomic_with_ifunc! {
+select_atomic! {
unsafe fn atomic_neg(dst: *mut u64) -> u64 {
- // SAFETY: the caller must uphold the safety contract.
- unsafe { atomic_update_kuser_cmpxchg64(dst, u64::wrapping_neg) }
+ |x| x.wrapping_neg()
}
fallback = atomic_neg_seqcst
}
@@ -310,7 +321,9 @@ mod tests {
fn kuser_helper_version() {
let version = __kuser_helper_version();
assert!(version >= 5, "{:?}", version);
- assert_eq!(version, unsafe { (KUSER_HELPER_VERSION as *const i32).read() });
+ assert_eq!(version, unsafe {
+ crate::utils::ptr::with_exposed_provenance::<i32>(KUSER_HELPER_VERSION).read()
+ });
}
test_atomic_int!(i64);
### external/vendor/portable-atomic/src/imp/atomic64/mod.rs
@@ -7,33 +7,35 @@ See README.md for details.
*/
// pre-v6 Arm Linux
-#[cfg(feature = "fallback")]
// Miri and Sanitizer do not support inline assembly.
#[cfg(all(
+ feature = "fallback",
target_arch = "arm",
not(any(miri, portable_atomic_sanitize_thread)),
any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
any(target_os = "linux", target_os = "android"),
- not(any(target_feature = "v6", portable_atomic_target_feature = "v6")),
+ any(test, not(any(target_feature = "v6", portable_atomic_target_feature = "v6"))),
not(portable_atomic_no_outline_atomics),
))]
-#[cfg_attr(portable_atomic_no_cfg_target_has_atomic, cfg(portable_atomic_no_atomic_64))]
-#[cfg_attr(not(portable_atomic_no_cfg_target_has_atomic), cfg(not(target_has_atomic = "64")))]
+#[cfg_attr(portable_atomic_no_cfg_target_has_atomic, cfg(any(test, portable_atomic_no_atomic_64)))]
+#[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(test, not(target_has_atomic = "64")))
+)]
pub(super) mod arm_linux;
// riscv32
// Miri and Sanitizer do not support inline assembly.
#[cfg(all(
target_arch = "riscv32",
not(any(miri, portable_atomic_sanitize_thread)),
- not(portable_atomic_no_asm),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
+ target_feature = "zacas",
+ portable_atomic_target_feature = "zacas",
all(
feature = "fallback",
not(portable_atomic_no_outline_atomics),
- any(test, portable_atomic_outline_atomics), // TODO(riscv): currently disabled by default
any(target_os = "linux", target_os = "android"),
),
),
### external/vendor/portable-atomic/src/imp/atomic64/riscv32.rs
@@ -17,88 +17,77 @@ this module and use fallback implementation instead.
Refs:
- RISC-V Instruction Set Manual
"Zacas" Extension for Atomic Compare-and-Swap (CAS) Instructions
- https://github.com/riscv/riscv-isa-manual/blob/riscv-isa-release-8b9dc50-2024-08-30/src/zacas.adoc
+ https://github.com/riscv/riscv-isa-manual/blob/riscv-isa-release-56e76be-2025-08-26/src/zacas.adoc
- RISC-V Atomics ABI Specification
- https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/draft-20240829-13bfa9f54634cb60d86b9b333e109f077805b4b3/riscv-atomic.adoc
+ https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/draft-20250812-301374e92976e298e676e7129a6212926b2299ce/riscv-atomic.adoc
-Generated asm:
-- riscv32imac (+experimental-zacas) https://godbolt.org/z/sq7f9W7rn
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
// TODO: merge duplicated code with atomic128/riscv64.rs
include!("macros.rs");
-#[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
-)))]
+#[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
#[path = "../fallback/outline_atomics.rs"]
mod fallback;
#[cfg(not(portable_atomic_no_outline_atomics))]
-#[cfg(any(test, portable_atomic_outline_atomics))] // TODO(riscv): currently disabled by default
-#[cfg(any(
- test,
- not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )),
-))]
+#[cfg(any(test, not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))))]
#[cfg(any(target_os = "linux", target_os = "android"))]
#[path = "../detect/riscv_linux.rs"]
mod detect;
-use core::{arch::asm, sync::atomic::Ordering};
+#[cfg(not(portable_atomic_no_asm))]
+use core::arch::asm;
+use core::sync::atomic::Ordering;
use crate::utils::{Pair, U64};
macro_rules! debug_assert_zacas {
() => {
- #[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )))]
+ #[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
{
- debug_assert!(detect::detect().has_zacas());
+ debug_assert!(detect::detect().zacas());
}
};
}
-// LLVM doesn't support `.option arch, +zabha` directive as of LLVM 19 because it is experimental.
-// So, we currently always using .4byte directive.
+// `.option arch, +zacas` directive requires LLVM 20, so we use .4byte directive for old LLVM.
// Note that `.insn <value>` directive requires LLVM 19.
// https://github.com/llvm/llvm-project/commit/2a086dce691e3cc34a2fc27f4fb255bb2cbbfac9
-// // https://github.com/riscv-non-isa/riscv-asm-manual/blob/ad0de8c004e29c9a7ac33cfd054f4d4f9392f2fb/src/asm-manual.adoc#arch
-// macro_rules! start_zacas {
-// () => {
-// ".option push\n.option arch, +zacas"
-// };
-// }
-// macro_rules! end_zacas {
-// () => {
-// ".option pop"
-// };
-// }
+// https://github.com/riscv-non-isa/riscv-asm-manual/blob/v0.0.1/src/asm-manual.adoc#arch
+#[cfg(not(portable_atomic_pre_llvm_20))]
+macro_rules! start_zacas {
+ () => {
+ ".option push\n.option arch, +zacas"
+ };
+}
+#[cfg(not(portable_atomic_pre_llvm_20))]
+macro_rules! end_zacas {
+ () => {
+ ".option pop"
+ };
+}
-// LLVM doesn't support `.option arch, +zabha` directive as of LLVM 19 because it is experimental.
-// So, we currently always using .4byte directive.
-// macro_rules! atomic_rmw_amocas_order {
-// ($op:ident, $order:ident) => {
-// atomic_rmw_amocas_order!($op, $order, failure = $order)
-// };
-// ($op:ident, $order:ident, failure = $failure:ident) => {
-// match $order {
-// Ordering::Relaxed => $op!("", ""),
-// Ordering::Acquire => $op!("", ".aq"),
-// Ordering::Release => $op!("", ".rl"),
-// Ordering::AcqRel => $op!("", ".aqrl"),
-// Ordering::SeqCst if $failure == Ordering::SeqCst => $op!("fence rw,rw", ".aqrl"),
-// Ordering::SeqCst => $op!("", ".aqrl"),
-// _ => unreachable!(),
-// }
-// };
-// }
+#[cfg(not(portable_atomic_pre_llvm_20))]
+macro_rules! atomic_rmw_amocas_order {
+ ($op:ident, $order:ident) => {
+ atomic_rmw_amocas_order!($op, $order, failure = $order)
+ };
+ ($op:ident, $order:ident, failure = $failure:ident) => {
+ match $order {
+ Ordering::Relaxed => $op!("", ""),
+ Ordering::Acquire => $op!("", ".aq"),
+ Ordering::Release => $op!("", ".rl"),
+ Ordering::AcqRel => $op!("", ".aqrl"),
+ Ordering::SeqCst if $failure == Ordering::SeqCst => $op!("fence rw,rw", ".aqrl"),
+ Ordering::SeqCst => $op!("", ".aqrl"),
+ _ => unreachable!(),
+ }
+ };
+}
+#[cfg(portable_atomic_pre_llvm_20)]
macro_rules! atomic_rmw_amocas_order_insn {
($op:ident, $order:ident) => {
atomic_rmw_amocas_order_insn!($op, $order, failure = $order)
@@ -117,16 +106,10 @@ macro_rules! atomic_rmw_amocas_order_insn {
}
// If zacas is available at compile-time, we can always use zacas_fn.
-#[cfg(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
-))]
+#[cfg(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))]
use self::atomic_load_zacas as atomic_load;
// Otherwise, we need to do run-time detection and can use zacas_fn only if zacas is available.
-#[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
-)))]
+#[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
#[inline]
unsafe fn atomic_load(src: *mut u64, order: Ordering) -> u64 {
fn_alias! {
@@ -144,7 +127,7 @@ unsafe fn atomic_load(src: *mut u64, order: Ordering) -> u64 {
match order {
Ordering::Relaxed => {
ifunc!(unsafe fn(src: *mut u64) -> u64 {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
atomic_load_zacas_relaxed
} else {
fallback::atomic_load_non_seqcst
@@ -153,7 +136,7 @@ unsafe fn atomic_load(src: *mut u64, order: Ordering) -> u64 {
}
Ordering::Acquire => {
ifunc!(unsafe fn(src: *mut u64) -> u64 {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
atomic_load_zacas_acquire
} else {
fallback::atomic_load_non_seqcst
@@ -162,7 +145,7 @@ unsafe fn atomic_load(src: *mut u64, order: Ordering) -> u64 {
}
Ordering::SeqCst => {
ifunc!(unsafe fn(src: *mut u64) -> u64 {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
atomic_load_zacas_seqcst
} else {
fallback::atomic_load_seqcst
@@ -181,23 +164,24 @@ unsafe fn atomic_load_zacas(src: *mut u64, order: Ordering) -> u64 {
// SAFETY: the caller must uphold the safety contract.
unsafe {
- // LLVM doesn't support `.option arch, +zabha` directive as of LLVM 19 because it is experimental.
- // So, we currently always using .4byte directive.
- // macro_rules! load {
- // ($fence:tt, $asm_order:tt) => {
- // asm!(
- // start_zacas!(),
- // $fence, // fence
- // concat!("amocas.d", $asm_order, " a2, a2, 0({src})"), // atomic { if *dst == a2:a3 { *dst = a2:a3 } else { a2:a3 = *dst } }
- // end_zacas!(),
- // src = in(reg) ptr_reg!(src),
- // inout("a2") 0_u32 => out_lo,
- // inout("a3") 0_u32 => out_hi,
- // options(nostack, preserves_flags),
- // )
- // };
- // }
- // atomic_rmw_amocas_order!(load, order);
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ macro_rules! load {
+ ($fence:tt, $asm_order:tt) => {
+ asm!(
+ start_zacas!(),
+ $fence, // fence
+ concat!("amocas.d", $asm_order, " a2, a2, 0({src})"), // atomic { if *dst == a2:a3 { *dst = a2:a3 } else { a2:a3 = *dst } }
+ end_zacas!(),
+ src = in(reg) ptr_reg!(src),
+ inout("a2") 0_u32 => out_lo,
+ inout("a3") 0_u32 => out_hi,
+ options(nostack, preserves_flags),
+ )
+ };
+ }
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ atomic_rmw_amocas_order!(load, order);
+ #[cfg(portable_atomic_pre_llvm_20)]
macro_rules! load {
($fence:tt, $insn_order:tt) => {
asm!(
@@ -211,6 +195,7 @@ unsafe fn atomic_load_zacas(src: *mut u64, order: Ordering) -> u64 {
)
};
}
+ #[cfg(portable_atomic_pre_llvm_20)]
atomic_rmw_amocas_order_insn!(load, order);
U64 { pair: Pair { lo: out_lo, hi: out_hi } }.whole
}
@@ -232,17 +217,11 @@ unsafe fn atomic_compare_exchange(
success: Ordering,
failure: Ordering,
) -> Result<u64, u64> {
- #[cfg(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- ))]
+ #[cfg(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))]
// SAFETY: the caller must uphold the safety contract.
// cfg guarantees that zacas instructions are available at compile-time.
let (prev, ok) = unsafe { atomic_compare_exchange_zacas(dst, old, new, success, failure) };
- #[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )))]
+ #[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
let (prev, ok) = {
fn_alias! {
// inline(never) is just a hint and also not strictly necessary
@@ -262,7 +241,7 @@ unsafe fn atomic_compare_exchange(
match order {
Ordering::Relaxed => {
ifunc!(unsafe fn(dst: *mut u64, old: u64, new: u64) -> (u64, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_relaxed_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -271,7 +250,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::Acquire => {
ifunc!(unsafe fn(dst: *mut u64, old: u64, new: u64) -> (u64, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_acquire_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -280,7 +259,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::Release => {
ifunc!(unsafe fn(dst: *mut u64, old: u64, new: u64) -> (u64, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_release_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -289,7 +268,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::AcqRel => {
ifunc!(unsafe fn(dst: *mut u64, old: u64, new: u64) -> (u64, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_acqrel_fn
} else {
fallback::atomic_compare_exchange_non_seqcst
@@ -298,7 +277,7 @@ unsafe fn atomic_compare_exchange(
}
Ordering::SeqCst => {
ifunc!(unsafe fn(dst: *mut u64, old: u64, new: u64) -> (u64, bool) {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_seqcst_fn
} else {
fallback::atomic_compare_exchange_seqcst
@@ -309,11 +288,7 @@ unsafe fn atomic_compare_exchange(
}
}
};
- if ok {
- Ok(prev)
- } else {
- Err(prev)
- }
+ if ok { Ok(prev) } else { Err(prev) }
}
#[inline]
unsafe fn atomic_compare_exchange_zacas(
@@ -332,27 +307,28 @@ unsafe fn atomic_compare_exchange_zacas(
// SAFETY: the caller must uphold the safety contract.
unsafe {
- // LLVM doesn't support `.option arch, +zabha` directive as of LLVM 19 because it is experimental.
- // So, we currently always using .4byte directive.
- // macro_rules! cmpxchg {
- // ($fence:tt, $asm_order:tt) => {
- // asm!(
- // start_zacas!(),
- // $fence, // fence
- // concat!("amocas.d", $asm_order, " a4, a2, 0({dst})"), // atomic { if *dst == a4:a5 { *dst = a2:a3 } else { a4:a5 = *dst } }
- // end_zacas!(),
- // dst = in(reg) ptr_reg!(dst),
- // // must be allocated to even/odd register pair
- // inout("a4") old.pair.lo => prev_lo,
- // inout("a5") old.pair.hi => prev_hi,
- // // must be allocated to even/odd register pair
- // in("a2") new.pair.lo,
- // in("a3") new.pair.hi,
- // options(nostack, preserves_flags),
- // )
- // };
- // }
- // atomic_rmw_amocas_order!(cmpxchg, order, failure = failure);
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ macro_rules! cmpxchg {
+ ($fence:tt, $asm_order:tt) => {
+ asm!(
+ start_zacas!(),
+ $fence, // fence
+ concat!("amocas.d", $asm_order, " a4, a2, 0({dst})"), // atomic { if *dst == a4:a5 { *dst = a2:a3 } else { a4:a5 = *dst } }
+ end_zacas!(),
+ dst = in(reg) ptr_reg!(dst),
+ // must be allocated to even/odd register pair
+ inout("a4") old.pair.lo => prev_lo,
+ inout("a5") old.pair.hi => prev_hi,
+ // must be allocated to even/odd register pair
+ in("a2") new.pair.lo,
+ in("a3") new.pair.hi,
+ options(nostack, preserves_flags),
+ )
+ };
+ }
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ atomic_rmw_amocas_order!(cmpxchg, order, failure = failure);
+ #[cfg(portable_atomic_pre_llvm_20)]
macro_rules! cmpxchg {
($fence:tt, $insn_order:tt) => {
asm!(
@@ -370,6 +346,7 @@ unsafe fn atomic_compare_exchange_zacas(
)
};
}
+ #[cfg(portable_atomic_pre_llvm_20)]
atomic_rmw_amocas_order_insn!(cmpxchg, order, failure = failure);
let prev = U64 { pair: Pair { lo: prev_lo, hi: prev_hi } }.whole;
(prev, prev == old.whole)
@@ -388,7 +365,7 @@ unsafe fn byte_wise_atomic_load(src: *const u64) -> u64 {
unsafe {
asm!(
"lw {out_lo}, ({src})", // atomic { out_lo = *src }
- "lw {out_hi}, 4({src})", // atomic { out_hi = *src.add(4) }
+ "lw {out_hi}, 4({src})", // atomic { out_hi = *src.byte_add(4) }
src = in(reg) ptr_reg!(src),
out_lo = out(reg) out_lo,
out_hi = out(reg) out_hi,
@@ -432,16 +409,10 @@ macro_rules! select_atomic_rmw {
}
}
// If zacas is available at compile-time, we can always use zacas_fn.
- #[cfg(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- ))]
+ #[cfg(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))]
use self::$zacas_fn as $name;
// Otherwise, we need to do run-time detection and can use zacas_fn only if zacas is available.
- #[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )))]
+ #[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
#[inline]
unsafe fn $name(dst: *mut u64 $(, $($arg)*)?, order: Ordering) $(-> $ret_ty)? {
fn_alias! {
@@ -461,7 +432,7 @@ macro_rules! select_atomic_rmw {
match order {
Ordering::Relaxed => {
ifunc!(unsafe fn(dst: *mut u64 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_relaxed_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -470,7 +441,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::Acquire => {
ifunc!(unsafe fn(dst: *mut u64 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_acquire_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -479,7 +450,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::Release => {
ifunc!(unsafe fn(dst: *mut u64 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_release_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -488,7 +459,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::AcqRel => {
ifunc!(unsafe fn(dst: *mut u64 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_acqrel_fn
} else {
fallback::$non_seqcst_fallback_fn
@@ -497,7 +468,7 @@ macro_rules! select_atomic_rmw {
}
Ordering::SeqCst => {
ifunc!(unsafe fn(dst: *mut u64 $(, $($arg)*)?) $(-> $ret_ty)? {
- if detect::detect().has_zacas() {
+ if detect::detect().zacas() {
zacas_seqcst_fn
} else {
fallback::$seqcst_fallback_fn
@@ -624,31 +595,22 @@ select_atomic_rmw! {
#[inline]
fn is_lock_free() -> bool {
- #[cfg(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- ))]
+ #[cfg(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"))]
{
// zacas is available at compile-time.
true
}
- #[cfg(not(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- )))]
+ #[cfg(not(any(target_feature = "zacas", portable_atomic_target_feature = "zacas")))]
{
- detect::detect().has_zacas()
+ detect::detect().zacas()
}
}
-const IS_ALWAYS_LOCK_FREE: bool = cfg!(any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
-));
+const IS_ALWAYS_LOCK_FREE: bool =
+ cfg!(any(target_feature = "zacas", portable_atomic_target_feature = "zacas"));
atomic64!(AtomicI64, i64, atomic_max, atomic_min);
atomic64!(AtomicU64, u64, atomic_umax, atomic_umin);
-#[allow(clippy::undocumented_unsafe_blocks, clippy::wildcard_imports)]
#[cfg(test)]
mod tests {
use super::*;
### external/vendor/portable-atomic/src/imp/avr.rs
@@ -18,8 +18,7 @@ Refs:
- atomic-maybe-uninit
https://github.com/taiki-e/atomic-maybe-uninit
-Generated asm:
-- avr https://godbolt.org/z/j49rYbj4d
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
use core::{arch::asm, cell::UnsafeCell, sync::atomic::Ordering};
@@ -49,8 +48,8 @@ macro_rules! atomic8 {
let out;
asm!(
"ld {out}, Z", // atomic { out = *Z }
- in("Z") src,
out = out(reg) out,
+ in("Z") src,
options(nostack, preserves_flags),
);
out
@@ -67,11 +66,30 @@ macro_rules! atomic8 {
unsafe {
asm!(
"st Z, {val}", // atomic { *Z = val }
- in("Z") dst,
val = in(reg) val,
+ in("Z") dst,
+ options(nostack, preserves_flags),
+ );
+ }
+ }
+
+ #[cfg(any(target_feature = "rmw", portable_atomic_target_feature = "rmw"))]
+ #[inline]
+ pub(crate) fn swap(&self, val: $value_type, _order: Ordering) -> $value_type {
+ let dst = self.v.get();
+ let out;
+ // SAFETY: any data races are prevented by atomic intrinsics and the raw
+ // pointer passed in is valid because we got it from a reference.
+ // cfg guarantee that the CPU supports RMW instructions.
+ unsafe {
+ asm!(
+ "xch Z, {val}", // atomic { _x = *Z; *Z = val; val = _x }
+ val = inout(reg) val => out,
+ in("Z") dst,
options(nostack, preserves_flags),
);
}
+ out
}
}
};
### external/vendor/portable-atomic/src/imp/core_atomic.rs
@@ -70,42 +70,168 @@ impl<T> AtomicPtr<T> {
}
#[cfg_attr(portable_atomic_no_cfg_target_has_atomic, cfg(not(portable_atomic_no_atomic_cas)))]
#[cfg_attr(not(portable_atomic_no_cfg_target_has_atomic), cfg(target_has_atomic = "ptr"))]
-impl<T> AtomicPtr<T> {
- #[inline]
- #[cfg_attr(
- any(all(debug_assertions, not(portable_atomic_no_track_caller)), miri),
- track_caller
- )]
- pub(crate) fn compare_exchange(
- &self,
- current: *mut T,
- new: *mut T,
- success: Ordering,
- failure: Ordering,
- ) -> Result<*mut T, *mut T> {
- crate::utils::assert_compare_exchange_ordering(success, failure); // for track_caller (compiler can omit double check)
- #[cfg(portable_atomic_no_stronger_failure_ordering)]
- let success = crate::utils::upgrade_success_ordering(success, failure);
- self.inner.compare_exchange(current, new, success, failure)
+items!({
+ impl<T> AtomicPtr<T> {
+ #[inline]
+ #[cfg_attr(
+ any(all(debug_assertions, not(portable_atomic_no_track_caller)), miri),
+ track_caller
+ )]
+ pub(crate) fn compare_exchange(
+ &self,
+ current: *mut T,
+ new: *mut T,
+ success: Ordering,
+ failure: Ordering,
+ ) -> Result<*mut T, *mut T> {
+ crate::utils::assert_compare_exchange_ordering(success, failure); // for track_caller (compiler can omit double check)
+ #[cfg(portable_atomic_no_stronger_failure_ordering)]
+ let success = crate::utils::upgrade_success_ordering(success, failure);
+ self.inner.compare_exchange(current, new, success, failure)
+ }
+ #[inline]
+ #[cfg_attr(
+ any(all(debug_assertions, not(portable_atomic_no_track_caller)), miri),
+ track_caller
+ )]
+ pub(crate) fn compare_exchange_weak(
+ &self,
+ current: *mut T,
+ new: *mut T,
+ success: Ordering,
+ failure: Ordering,
+ ) -> Result<*mut T, *mut T> {
+ crate::utils::assert_compare_exchange_ordering(success, failure); // for track_caller (compiler can omit double check)
+ #[cfg(portable_atomic_no_stronger_failure_ordering)]
+ let success = crate::utils::upgrade_success_ordering(success, failure);
+ self.inner.compare_exchange_weak(current, new, success, failure)
+ }
}
- #[inline]
- #[cfg_attr(
- any(all(debug_assertions, not(portable_atomic_no_track_caller)), miri),
- track_caller
- )]
- pub(crate) fn compare_exchange_weak(
- &self,
- current: *mut T,
- new: *mut T,
- success: Ordering,
- failure: Ordering,
- ) -> Result<*mut T, *mut T> {
- crate::utils::assert_compare_exchange_ordering(success, failure); // for track_caller (compiler can omit double check)
- #[cfg(portable_atomic_no_stronger_failure_ordering)]
- let success = crate::utils::upgrade_success_ordering(success, failure);
- self.inner.compare_exchange_weak(current, new, success, failure)
+ // Ideally, we would always use AtomicPtr::fetch_* since it is strict-provenance
+ // compatible, but it requires 1.91+. So, for now emulate it only on cfg(miri).
+ // Code using AtomicUsize::fetch_* via casts is still permissive-provenance
+ // compatible and is sound.
+ #[cfg(portable_atomic_no_strict_provenance_atomic_ptr)]
+ impl<T> AtomicPtr<T> {
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_byte_add(&self, val: usize, order: Ordering) -> *mut T {
+ #[cfg(miri)]
+ {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ self.fetch_update_(order, |x| x.with_addr(x.addr().wrapping_add(val)))
+ }
+ #[cfg(not(miri))]
+ {
+ crate::utils::ptr::with_exposed_provenance_mut(
+ self.as_atomic_usize().fetch_add(val, order),
+ )
+ }
+ }
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_byte_sub(&self, val: usize, order: Ordering) -> *mut T {
+ #[cfg(miri)]
+ {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ self.fetch_update_(order, |x| x.with_addr(x.addr().wrapping_sub(val)))
+ }
+ #[cfg(not(miri))]
+ {
+ crate::utils::ptr::with_exposed_provenance_mut(
+ self.as_atomic_usize().fetch_sub(val, order),
+ )
+ }
+ }
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_or(&self, val: usize, order: Ordering) -> *mut T {
+ #[cfg(miri)]
+ {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ self.fetch_update_(order, |x| x.with_addr(x.addr() | val))
+ }
+ #[cfg(not(miri))]
+ {
+ crate::utils::ptr::with_exposed_provenance_mut(
+ self.as_atomic_usize().fetch_or(val, order),
+ )
+ }
+ }
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_and(&self, val: usize, order: Ordering) -> *mut T {
+ #[cfg(miri)]
+ {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ self.fetch_update_(order, |x| x.with_addr(x.addr() & val))
+ }
+ #[cfg(not(miri))]
+ {
+ crate::utils::ptr::with_exposed_provenance_mut(
+ self.as_atomic_usize().fetch_and(val, order),
+ )
+ }
+ }
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_xor(&self, val: usize, order: Ordering) -> *mut T {
+ #[cfg(miri)]
+ {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ self.fetch_update_(order, |x| x.with_addr(x.addr() ^ val))
+ }
+ #[cfg(not(miri))]
+ {
+ crate::utils::ptr::with_exposed_provenance_mut(
+ self.as_atomic_usize().fetch_xor(val, order),
+ )
+ }
+ }
+ #[cfg(miri)]
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ fn fetch_update_<F>(&self, order: Ordering, mut f: F) -> *mut T
+ where
+ F: FnMut(*mut T) -> *mut T,
+ {
+ // This is a private function and all instances of `f` only operate on the value
+ // loaded, so there is no need to synchronize the first load/failed CAS.
+ let mut prev = self.load(Ordering::Relaxed);
+ loop {
+ let next = f(prev);
+ match self.compare_exchange_weak(prev, next, order, Ordering::Relaxed) {
+ Ok(x) => return x,
+ Err(next_prev) => prev = next_prev,
+ }
+ }
+ }
+ #[cfg(not(miri))]
+ #[inline(always)]
+ fn as_atomic_usize(&self) -> &AtomicUsize {
+ static_assert!(
+ core::mem::size_of::<AtomicPtr<()>>() == core::mem::size_of::<AtomicUsize>()
+ );
+ static_assert!(
+ core::mem::align_of::<AtomicPtr<()>>() == core::mem::align_of::<AtomicUsize>()
+ );
+ // SAFETY: AtomicPtr and AtomicUsize have the same layout,
+ // and both access data in the same way.
+ unsafe { &*(self as *const Self as *const AtomicUsize) }
+ }
}
-}
+ #[cfg(not(all(
+ any(target_arch = "x86", target_arch = "x86_64"),
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ )))]
+ impl_default_bit_opts!(AtomicPtr, usize);
+});
impl<T> core::ops::Deref for AtomicPtr<T> {
type Target = core::sync::atomic::AtomicPtr<T>;
#[inline]
@@ -123,23 +249,6 @@ macro_rules! atomic_int {
// Prevent RefUnwindSafe from being propagated from the std atomic type. See NotRefUnwindSafe for more.
_not_ref_unwind_safe: PhantomData<NotRefUnwindSafe>,
}
- #[cfg_attr(
- portable_atomic_no_cfg_target_has_atomic,
- cfg(not(portable_atomic_no_atomic_cas))
- )]
- #[cfg_attr(not(portable_atomic_no_cfg_target_has_atomic), cfg(target_has_atomic = "ptr"))]
- impl_default_no_fetch_ops!($atomic_type, $int_type);
- #[cfg(not(all(
- any(target_arch = "x86", target_arch = "x86_64"),
- not(any(miri, portable_atomic_sanitize_thread)),
- any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
- )))]
- #[cfg_attr(
- portable_atomic_no_cfg_target_has_atomic,
- cfg(not(portable_atomic_no_atomic_cas))
- )]
- #[cfg_attr(not(portable_atomic_no_cfg_target_has_atomic), cfg(target_has_atomic = "ptr"))]
- impl_default_bit_opts!($atomic_type, $int_type);
impl $atomic_type {
#[inline]
pub(crate) const fn new(v: $int_type) -> Self {
@@ -195,195 +304,204 @@ macro_rules! atomic_int {
cfg(not(portable_atomic_no_atomic_cas))
)]
#[cfg_attr(not(portable_atomic_no_cfg_target_has_atomic), cfg(target_has_atomic = "ptr"))]
- impl $atomic_type {
- #[inline]
- #[cfg_attr(
- any(all(debug_assertions, not(portable_atomic_no_track_caller)), miri),
- track_caller
- )]
- pub(crate) fn compare_exchange(
- &self,
- current: $int_type,
- new: $int_type,
- success: Ordering,
- failure: Ordering,
- ) -> Result<$int_type, $int_type> {
- crate::utils::assert_compare_exchange_ordering(success, failure); // for track_caller (compiler can omit double check)
- #[cfg(portable_atomic_no_stronger_failure_ordering)]
- let success = crate::utils::upgrade_success_ordering(success, failure);
- self.inner.compare_exchange(current, new, success, failure)
- }
- #[inline]
- #[cfg_attr(
- any(all(debug_assertions, not(portable_atomic_no_track_caller)), miri),
- track_caller
- )]
- pub(crate) fn compare_exchange_weak(
- &self,
- current: $int_type,
- new: $int_type,
- success: Ordering,
- failure: Ordering,
- ) -> Result<$int_type, $int_type> {
- crate::utils::assert_compare_exchange_ordering(success, failure); // for track_caller (compiler can omit double check)
- #[cfg(portable_atomic_no_stronger_failure_ordering)]
- let success = crate::utils::upgrade_success_ordering(success, failure);
- self.inner.compare_exchange_weak(current, new, success, failure)
- }
- #[allow(dead_code)]
- #[inline]
- #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
- fn fetch_update_<F>(&self, order: Ordering, mut f: F) -> $int_type
- where
- F: FnMut($int_type) -> $int_type,
- {
- // This is a private function and all instances of `f` only operate on the value
- // loaded, so there is no need to synchronize the first load/failed CAS.
- let mut prev = self.load(Ordering::Relaxed);
- loop {
- let next = f(prev);
- match self.compare_exchange_weak(prev, next, order, Ordering::Relaxed) {
- Ok(x) => return x,
- Err(next_prev) => prev = next_prev,
- }
+ items!({
+ impl_default_no_fetch_ops!($atomic_type, $int_type);
+ impl $atomic_type {
+ #[inline]
+ #[cfg_attr(
+ any(all(debug_assertions, not(portable_atomic_no_track_caller)), miri),
+ track_caller
+ )]
+ pub(crate) fn compare_exchange(
+ &self,
+ current: $int_type,
+ new: $int_type,
+ success: Ordering,
+ failure: Ordering,
+ ) -> Result<$int_type, $int_type> {
+ crate::utils::assert_compare_exchange_ordering(success, failure); // for track_caller (compiler can omit double check)
+ #[cfg(portable_atomic_no_stronger_failure_ordering)]
+ let success = crate::utils::upgrade_success_ordering(success, failure);
+ self.inner.compare_exchange(current, new, success, failure)
}
- }
- #[inline]
- #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
- pub(crate) fn fetch_max(&self, val: $int_type, order: Ordering) -> $int_type {
- #[cfg(not(portable_atomic_no_atomic_min_max))]
+ #[inline]
+ #[cfg_attr(
+ any(all(debug_assertions, not(portable_atomic_no_track_caller)), miri),
+ track_caller
+ )]
+ pub(crate) fn compare_exchange_weak(
+ &self,
+ current: $int_type,
+ new: $int_type,
+ success: Ordering,
+ failure: Ordering,
+ ) -> Result<$int_type, $int_type> {
+ crate::utils::assert_compare_exchange_ordering(success, failure); // for track_caller (compiler can omit double check)
+ #[cfg(portable_atomic_no_stronger_failure_ordering)]
+ let success = crate::utils::upgrade_success_ordering(success, failure);
+ self.inner.compare_exchange_weak(current, new, success, failure)
+ }
+ #[allow(dead_code)]
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ fn fetch_update_<F>(&self, order: Ordering, mut f: F) -> $int_type
+ where
+ F: FnMut($int_type) -> $int_type,
{
- #[cfg(any(
- all(
- any(target_arch = "aarch64", target_arch = "arm64ec"),
- any(target_feature = "lse", portable_atomic_target_feature = "lse"),
- ),
- all(
- target_arch = "arm",
- not(any(
- target_feature = "v6",
- portable_atomic_target_feature = "v6",
- )),
- ),
- target_arch = "mips",
- target_arch = "mips32r6",
- target_arch = "mips64",
- target_arch = "mips64r6",
- target_arch = "powerpc",
- target_arch = "powerpc64",
- ))]
- {
- // HACK: the following operations are currently broken (at least on qemu-user):
- // - aarch64's `AtomicI{8,16}::fetch_{max,min}` (release mode + lse)
- // - armv5te's `Atomic{I,U}{8,16}::fetch_{max,min}`
- // - mips's `AtomicI8::fetch_{max,min}` (release mode)
- // - mipsel's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
- // - mips64's `AtomicI8::fetch_{max,min}` (release mode)
- // - mips64el's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
- // - powerpc's `AtomicI{8,16}::fetch_{max,min}`
- // - powerpc64's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
- // - powerpc64le's `AtomicU{8,16}::fetch_{max,min}` (release mode + fat LTO)
- // See also:
- // https://github.com/llvm/llvm-project/issues/61880
- // https://github.com/llvm/llvm-project/issues/61881
- // https://github.com/llvm/llvm-project/issues/61882
- // https://github.com/taiki-e/portable-atomic/issues/2
- // https://github.com/rust-lang/rust/issues/100650
- if core::mem::size_of::<$int_type>() <= 2 {
- return self.fetch_update_(order, |x| core::cmp::max(x, val));
+ // This is a private function and all instances of `f` only operate on the value
+ // loaded, so there is no need to synchronize the first load/failed CAS.
+ let mut prev = self.load(Ordering::Relaxed);
+ loop {
+ let next = f(prev);
+ match self.compare_exchange_weak(prev, next, order, Ordering::Relaxed) {
+ Ok(x) => return x,
+ Err(next_prev) => prev = next_prev,
}
}
- self.inner.fetch_max(val, order)
}
- #[cfg(portable_atomic_no_atomic_min_max)]
- {
- self.fetch_update_(order, |x| core::cmp::max(x, val))
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_max(&self, val: $int_type, order: Ordering) -> $int_type {
+ #[cfg(not(portable_atomic_no_atomic_min_max))]
+ {
+ #[cfg(any(
+ all(
+ any(target_arch = "aarch64", target_arch = "arm64ec"),
+ any(
+ target_feature = "lse",
+ portable_atomic_target_feature = "lse",
+ ),
+ ),
+ all(
+ target_arch = "arm",
+ not(any(
+ target_feature = "v6",
+ portable_atomic_target_feature = "v6",
+ )),
+ ),
+ target_arch = "mips",
+ target_arch = "mips32r6",
+ target_arch = "mips64",
+ target_arch = "mips64r6",
+ target_arch = "powerpc",
+ target_arch = "powerpc64",
+ ))]
+ {
+ // HACK: the following operations are currently broken (at least on qemu-user):
+ // - aarch64's `AtomicI{8,16}::fetch_{max,min}` (release mode + lse)
+ // - armv5te's `Atomic{I,U}{8,16}::fetch_{max,min}`
+ // - mips's `AtomicI8::fetch_{max,min}` (release mode)
+ // - mipsel's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
+ // - mips64's `AtomicI8::fetch_{max,min}` (release mode)
+ // - mips64el's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
+ // - powerpc's `AtomicI{8,16}::fetch_{max,min}`
+ // - powerpc64's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
+ // - powerpc64le's `AtomicU{8,16}::fetch_{max,min}` (release mode + fat LTO)
+ // See also:
+ // https://github.com/llvm/llvm-project/issues/61880
+ // https://github.com/llvm/llvm-project/issues/61881
+ // https://github.com/llvm/llvm-project/issues/61882
+ // https://github.com/taiki-e/portable-atomic/issues/2
+ // https://github.com/rust-lang/rust/issues/100650
+ if core::mem::size_of::<$int_type>() <= 2 {
+ return self.fetch_update_(order, |x| core::cmp::max(x, val));
+ }
+ }
+ self.inner.fetch_max(val, order)
+ }
+ #[cfg(portable_atomic_no_atomic_min_max)]
+ {
+ self.fetch_update_(order, |x| core::cmp::max(x, val))
+ }
}
- }
- #[inline]
- #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
- pub(crate) fn fetch_min(&self, val: $int_type, order: Ordering) -> $int_type {
- #[cfg(not(portable_atomic_no_atomic_min_max))]
- {
- #[cfg(any(
- all(
- any(target_arch = "aarch64", target_arch = "arm64ec"),
- any(target_feature = "lse", portable_atomic_target_feature = "lse"),
- ),
- all(
- target_arch = "arm",
- not(any(
- target_feature = "v6",
- portable_atomic_target_feature = "v6",
- )),
- ),
- target_arch = "mips",
- target_arch = "mips32r6",
- target_arch = "mips64",
- target_arch = "mips64r6",
- target_arch = "powerpc",
- target_arch = "powerpc64",
- ))]
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_min(&self, val: $int_type, order: Ordering) -> $int_type {
+ #[cfg(not(portable_atomic_no_atomic_min_max))]
{
- // HACK: the following operations are currently broken (at least on qemu-user):
- // - aarch64's `AtomicI{8,16}::fetch_{max,min}` (release mode + lse)
- // - armv5te's `Atomic{I,U}{8,16}::fetch_{max,min}`
- // - mips's `AtomicI8::fetch_{max,min}` (release mode)
- // - mipsel's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
- // - mips64's `AtomicI8::fetch_{max,min}` (release mode)
- // - mips64el's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
- // - powerpc's `AtomicI{8,16}::fetch_{max,min}`
- // - powerpc64's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
- // - powerpc64le's `AtomicU{8,16}::fetch_{max,min}` (release mode + fat LTO)
- // See also:
- // https://github.com/llvm/llvm-project/issues/61880
- // https://github.com/llvm/llvm-project/issues/61881
- // https://github.com/llvm/llvm-project/issues/61882
- // https://github.com/taiki-e/portable-atomic/issues/2
- // https://github.com/rust-lang/rust/issues/100650
- if core::mem::size_of::<$int_type>() <= 2 {
- return self.fetch_update_(order, |x| core::cmp::min(x, val));
+ #[cfg(any(
+ all(
+ any(target_arch = "aarch64", target_arch = "arm64ec"),
+ any(
+ target_feature = "lse",
+ portable_atomic_target_feature = "lse",
+ ),
+ ),
+ all(
+ target_arch = "arm",
+ not(any(
+ target_feature = "v6",
+ portable_atomic_target_feature = "v6",
+ )),
+ ),
+ target_arch = "mips",
+ target_arch = "mips32r6",
+ target_arch = "mips64",
+ target_arch = "mips64r6",
+ target_arch = "powerpc",
+ target_arch = "powerpc64",
+ ))]
+ {
+ // HACK: the following operations are currently broken (at least on qemu-user):
+ // - aarch64's `AtomicI{8,16}::fetch_{max,min}` (release mode + lse)
+ // - armv5te's `Atomic{I,U}{8,16}::fetch_{max,min}`
+ // - mips's `AtomicI8::fetch_{max,min}` (release mode)
+ // - mipsel's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
+ // - mips64's `AtomicI8::fetch_{max,min}` (release mode)
+ // - mips64el's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
+ // - powerpc's `AtomicI{8,16}::fetch_{max,min}`
+ // - powerpc64's `AtomicI{8,16}::fetch_{max,min}` (debug mode, at least)
+ // - powerpc64le's `AtomicU{8,16}::fetch_{max,min}` (release mode + fat LTO)
+ // See also:
+ // https://github.com/llvm/llvm-project/issues/61880
+ // https://github.com/llvm/llvm-project/issues/61881
+ // https://github.com/llvm/llvm-project/issues/61882
+ // https://github.com/taiki-e/portable-atomic/issues/2
+ // https://github.com/rust-lang/rust/issues/100650
+ if core::mem::size_of::<$int_type>() <= 2 {
+ return self.fetch_update_(order, |x| core::cmp::min(x, val));
+ }
}
+ self.inner.fetch_min(val, order)
+ }
+ #[cfg(portable_atomic_no_atomic_min_max)]
+ {
+ self.fetch_update_(order, |x| core::cmp::min(x, val))
}
- self.inner.fetch_min(val, order)
}
- #[cfg(portable_atomic_no_atomic_min_max)]
- {
- self.fetch_update_(order, |x| core::cmp::min(x, val))
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_not(&self, order: Ordering) -> $int_type {
+ self.fetch_xor(!0, order)
+ }
+ // TODO: provide asm-based implementation on AArch64 without FEAT_LSE, Armv7, RISC-V, etc.
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_neg(&self, order: Ordering) -> $int_type {
+ self.fetch_update_(order, $int_type::wrapping_neg)
}
- }
- #[inline]
- #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
- pub(crate) fn fetch_not(&self, order: Ordering) -> $int_type {
- self.fetch_xor(!0, order)
- }
- #[cfg(not(all(
- any(target_arch = "x86", target_arch = "x86_64"),
- not(any(miri, portable_atomic_sanitize_thread)),
- any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
- )))]
- #[inline]
- #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
- pub(crate) fn not(&self, order: Ordering) {
- self.fetch_not(order);
- }
- // TODO: provide asm-based implementation on AArch64 without FEAT_LSE, Armv7, RISC-V, etc.
- #[inline]
- #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
- pub(crate) fn fetch_neg(&self, order: Ordering) -> $int_type {
- self.fetch_update_(order, $int_type::wrapping_neg)
}
#[cfg(not(all(
any(target_arch = "x86", target_arch = "x86_64"),
not(any(miri, portable_atomic_sanitize_thread)),
any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
)))]
- #[inline]
- #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
- pub(crate) fn neg(&self, order: Ordering) {
- self.fetch_neg(order);
- }
- }
+ items!({
+ impl_default_bit_opts!($atomic_type, $int_type);
+ impl $atomic_type {
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn not(&self, order: Ordering) {
+ self.fetch_not(order);
+ }
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn neg(&self, order: Ordering) {
+ self.fetch_neg(order);
+ }
+ }
+ });
+ });
impl core::ops::Deref for $atomic_type {
type Target = core::sync::atomic::$atomic_type;
#[inline]
@@ -419,13 +537,7 @@ atomic_int!(AtomicU32, u32);
not(any(target_pointer_width = "16", target_pointer_width = "32")),
))
)]
-atomic_int!(AtomicI64, i64);
-#[cfg_attr(portable_atomic_no_cfg_target_has_atomic, cfg(not(portable_atomic_no_atomic_64)))]
-#[cfg_attr(
- not(portable_atomic_no_cfg_target_has_atomic),
- cfg(any(
- target_has_atomic = "64",
- not(any(target_pointer_width = "16", target_pointer_width = "32")),
- ))
-)]
-atomic_int!(AtomicU64, u64);
+items!({
+ atomic_int!(AtomicI64, i64);
+ atomic_int!(AtomicU64, u64);
+});
### external/vendor/portable-atomic/src/imp/detect/README.md
@@ -6,27 +6,33 @@ Here is the table of targets that support run-time CPU feature detection and the
| target_arch | target_os/target_env | instruction/API | features | note |
| ----------- | -------------------- | --------------- | -------- | ---- |
-| x86_64 | all (except for sgx) | cpuid | all | Enabled by default |
-| aarch64 | linux | getauxval | all | Only enabled by default on `*-linux-gnu*` and `*-linux-{musl,ohos,uclibc}*` with dynamic linking enabled (musl is static linking by default). (dlsym is used by default if needed for compatibility with older versions) |
-| aarch64 | android | getauxval | all | Enabled by default |
-| aarch64 | freebsd | elf_aux_info | lse, lse2 | Enabled by default |
-| aarch64 | netbsd | sysctlbyname | all | Enabled by default |
-| aarch64 | openbsd | sysctl | all | Enabled by default |
-| aarch64 | macos/ios/tvos/watchos/visionos | sysctlbyname | all | Currently only used in tests (see [aarch64_apple.rs](aarch64_apple.rs)). |
-| aarch64 | illumos | getisax | lse, lse2 | Disabled by default |
-| aarch64/arm64ec | windows | IsProcessorFeaturePresent | lse | Enabled by default |
-| aarch64 | fuchsia | zx_system_get_features | lse | Enabled by default |
-| riscv32/riscv64 | linux/android | riscv_hwprobe | all | Disabled by default |
-| powerpc64 | linux | getauxval | all | Only enabled by default on `*-linux-{gnu,musl,ohos,uclibc}*` with dynamic linking enabled (musl is static linking by default). (dlsym is used by default if needed for compatibility with older versions) |
-| powerpc64 | freebsd | elf_aux_info | all | Enabled by default (dlsym is used by default for compatibility with older versions) |
-| powerpc64 | openbsd | elf_aux_info | all | Enabled by default (dlsym is used by default for compatibility with older versions) |
+| x86_64 | all (except for sgx) | cpuid | all | Enabled by default |
+| aarch64 | linux (gnu/ohos/uclibc) | getauxval | all | Enabled by default (dlsym is used by default if needed for compatibility with older versions) |
+| aarch64 | linux (musl) | getauxval | all | Only enabled by default when dynamic linking or `std` feature enabled (both disabled by default, see [auxv.rs](auxv.rs)) |
+| aarch64 | android | getauxval | all | Enabled by default |
+| aarch64 | freebsd | elf_aux_info | all | Enabled by default |
+| aarch64 | netbsd | sysctlbyname | all | Enabled by default |
+| aarch64 | openbsd | sysctl | all | Enabled by default |
+| aarch64 | macos/ios/tvos/watchos/visionos | sysctlbyname | all | Currently only used in tests (see [aarch64_apple.rs](aarch64_apple.rs)) |
+| aarch64 | illumos | getisax | lse, lse2 | Disabled by default (see [aarch64_illumos.rs](aarch64_illumos.rs)) |
+| aarch64/arm64ec | windows | IsProcessorFeaturePresent | lse | Enabled by default |
+| aarch64 | fuchsia | zx_system_get_features | lse | Enabled by default |
+| riscv32/riscv64 | linux/android | riscv_hwprobe | all | Enabled by default |
+| powerpc64 | linux (gnu/ohos/uclibc) | getauxval | all | Enabled by default (dlsym is used by default if needed for compatibility with older versions) |
+| powerpc64 | linux (musl) | getauxval | all | Only enabled by default when dynamic linking or `std` feature enabled (both disabled by default, see [auxv.rs](auxv.rs)) |
+| powerpc64 | freebsd | elf_aux_info | all | Enabled by default (dlsym is used by default for compatibility with older versions) |
+| powerpc64 | openbsd | elf_aux_info | all | Enabled by default (dlsym is used by default for compatibility with older versions) |
+| powerpc64 | aix | getsystemcfg | all | Requires LLVM 20+. Disabled by default (see [powerpc64_aix.rs](powerpc64_aix.rs)) |
Run-time detection is enabled by default on most targets and can be disabled with `--cfg portable_atomic_no_outline_atomics`.
-On some targets, run-time detection is disabled by default mainly for incomplete build environments, and can be enabled by `--cfg portable_atomic_outline_atomics`. (When both cfg are enabled, `*_no_*` cfg is preferred.)
+On some targets, run-time detection is disabled by default mainly for compatibility with incomplete build environments or support for it is experimental, and can be enabled by `--cfg portable_atomic_outline_atomics`. (When both cfg are enabled, `*_no_*` cfg is preferred.)
+
+`dlsym` usually not working with static linking, so detection using implementations that use `dlsym` for compatibility will be disabled if static linking is enabled.
+You can use `--cfg portable_atomic_outline_atomics` to force the use of non-`dlsym` implementations and enable run-time detection in such an environment.
For targets not included in the above table, run-time detection is always disabled and works the same as when `--cfg portable_atomic_no_outline_atomics` is set.
See [auxv.rs](auxv.rs) module-level comments for more details on Linux/Android/FreeBSD/OpenBSD.
-See also [docs on `portable_atomic_no_outline_atomics`](https://github.com/taiki-e/portable-atomic/blob/HEAD/README.md#optional-cfg-no-outline-atomics) in the top-level readme.
+See also [docs about `portable_atomic_no_outline_atomics` cfg](https://github.com/taiki-e/portable-atomic/blob/HEAD/README.md#optional-cfg-no-outline-atomics) in the top-level readme.
### external/vendor/portable-atomic/src/imp/detect/aarch64_aa64reg.rs
@@ -9,8 +9,8 @@ Run-time detection on OpenBSD by is_aarch64_feature_detected is supported on Rus
https://github.com/rust-lang/stdarch/pull/1374
Refs:
-- https://developer.arm.com/documentation/ddi0601/2024-06/AArch64-Registers
-- https://github.com/torvalds/linux/blob/v6.11/Documentation/arch/arm64/cpu-feature-registers.rst
+- https://developer.arm.com/documentation/ddi0601/2025-06/AArch64-Registers
+- https://github.com/torvalds/linux/blob/v6.16/Documentation/arch/arm64/cpu-feature-registers.rst
- https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/std_detect/src/detect/os/aarch64.rs
Supported platforms:
@@ -31,9 +31,10 @@ For now, this module is only used on NetBSD/OpenBSD.
On Linux/Android/FreeBSD, we use auxv.rs and this module is test-only because:
- On Linux/Android, this approach requires a higher kernel version than Rust supports,
- and also does not work with qemu-user (as of 7.2) and Valgrind (as of 3.19).
+ and also does not work with qemu-user (as of QEMU 7.2) and Valgrind (as of Valgrind 3.24).
(Looking into HWCAP_CPUID in auxvec, it appears that Valgrind is setting it
to false correctly, but qemu-user is setting it to true.)
+ - qemu-user issue seem to be fixed as of QEMU 9.2.
- On FreeBSD, this approach does not work on FreeBSD 12 on QEMU (confirmed on
FreeBSD 12.{2,3,4}), and we got SIGILL (worked on FreeBSD 13 and 14).
*/
@@ -44,31 +45,68 @@ include!("common.rs");
struct AA64Reg {
aa64isar0: u64,
aa64isar1: u64,
+ #[cfg(test)]
+ aa64isar3: u64,
aa64mmfr2: u64,
}
#[cold]
fn _detect(info: &mut CpuInfo) {
- let AA64Reg { aa64isar0, aa64isar1, aa64mmfr2 } = imp::aa64reg();
+ let AA64Reg {
+ aa64isar0,
+ aa64isar1,
+ #[cfg(test)]
+ aa64isar3,
+ aa64mmfr2,
+ } = imp::aa64reg();
// ID_AA64ISAR0_EL1, AArch64 Instruction Set Attribute Register 0
- // https://developer.arm.com/documentation/ddi0601/2024-06/AArch64-Registers/ID-AA64ISAR0-EL1--AArch64-Instruction-Set-Attribute-Register-0
+ // https://developer.arm.com/documentation/ddi0601/2025-06/AArch64-Registers/ID-AA64ISAR0-EL1--AArch64-Instruction-Set-Attribute-Register-0
+ // Atomic, bits [23:20]
+ // > FEAT_LSE implements the functionality identified by the value 0b0010.
+ // > FEAT_LSE128 implements the functionality identified by the value 0b0011.
+ // > From Armv8.1, the value 0b0000 is not permitted.
let atomic = extract(aa64isar0, 23, 20);
if atomic >= 0b0010 {
- info.set(CpuInfo::HAS_LSE);
+ info.set(CpuInfoFlag::lse);
if atomic >= 0b0011 {
- info.set(CpuInfo::HAS_LSE128);
+ info.set(CpuInfoFlag::lse128);
}
}
// ID_AA64ISAR1_EL1, AArch64 Instruction Set Attribute Register 1
- // https://developer.arm.com/documentation/ddi0601/2024-06/AArch64-Registers/ID-AA64ISAR1-EL1--AArch64-Instruction-Set-Attribute-Register-1
- if extract(aa64isar1, 23, 20) >= 0b0011 {
- info.set(CpuInfo::HAS_RCPC3);
+ // https://developer.arm.com/documentation/ddi0601/2025-06/AArch64-Registers/ID-AA64ISAR1-EL1--AArch64-Instruction-Set-Attribute-Register-1
+ // LRCPC, bits [23:20]
+ // > FEAT_LRCPC implements the functionality identified by the value 0b0001.
+ // > FEAT_LRCPC2 implements the functionality identified by the value 0b0010.
+ // > FEAT_LRCPC3 implements the functionality identified by the value 0b0011.
+ // > From Armv8.3, the value 0b0000 is not permitted.
+ // > From Armv8.4, the value 0b0001 is not permitted.
+ let lrcpc = extract(aa64isar1, 23, 20);
+ if lrcpc >= 0b0011 {
+ info.set(CpuInfoFlag::rcpc3);
+ }
+ #[cfg(test)]
+ if lrcpc >= 0b0001 {
+ info.set(CpuInfoFlag::rcpc);
+ if lrcpc >= 0b0010 {
+ info.set(CpuInfoFlag::rcpc2);
+ }
+ }
+ // ID_AA64ISAR3_EL1, AArch64 Instruction Set Attribute Register 3
+ // https://developer.arm.com/documentation/ddi0601/2025-06/AArch64-Registers/ID-AA64ISAR3-EL1--AArch64-Instruction-Set-Attribute-Register-3
+ // LSFE, bits [19:16]
+ // > FEAT_LSFE implements the functionality identified by the value 0b0001
+ #[cfg(test)]
+ if extract(aa64isar3, 19, 16) >= 0b0001 {
+ info.set(CpuInfoFlag::lsfe);
}
// ID_AA64MMFR2_EL1, AArch64 Memory Model Feature Register 2
- // https://developer.arm.com/documentation/ddi0601/2024-06/AArch64-Registers/ID-AA64MMFR2-EL1--AArch64-Memory-Model-Feature-Register-2
+ // https://developer.arm.com/documentation/ddi0601/2025-06/AArch64-Registers/ID-AA64MMFR2-EL1--AArch64-Memory-Model-Feature-Register-2
+ // AT, bits [35:32]
+ // > FEAT_LSE2 implements the functionality identified by the value 0b0001.
+ // > From Armv8.4, the value 0b0000 is not permitted.
if extract(aa64mmfr2, 35, 32) >= 0b0001 {
- info.set(CpuInfo::HAS_LSE2);
+ info.set(CpuInfoFlag::lse2);
}
}
@@ -92,23 +130,45 @@ mod imp {
unsafe {
let aa64isar0: u64;
asm!(
- "mrs {0}, ID_AA64ISAR0_EL1",
+ "mrs {}, ID_AA64ISAR0_EL1",
out(reg) aa64isar0,
options(pure, nomem, nostack, preserves_flags),
);
let aa64isar1: u64;
asm!(
- "mrs {0}, ID_AA64ISAR1_EL1",
+ "mrs {}, ID_AA64ISAR1_EL1",
out(reg) aa64isar1,
options(pure, nomem, nostack, preserves_flags),
);
+ #[cfg(test)]
+ #[cfg(not(portable_atomic_pre_llvm_18))]
+ let aa64isar3: u64;
+ // ID_AA64ISAR3_EL1 is only recognized on LLVM 18+.
+ // https://github.com/llvm/llvm-project/commit/17baba9fa2728b1b1134f9dccb9318debd5a9a1b
+ #[cfg(test)]
+ #[cfg(not(portable_atomic_pre_llvm_18))]
+ asm!(
+ "mrs {}, ID_AA64ISAR3_EL1",
+ out(reg) aa64isar3,
+ options(pure, nomem, nostack, preserves_flags),
+ );
let aa64mmfr2: u64;
asm!(
- "mrs {0}, ID_AA64MMFR2_EL1",
+ "mrs {}, ID_AA64MMFR2_EL1",
out(reg) aa64mmfr2,
options(pure, nomem, nostack, preserves_flags),
);
- AA64Reg { aa64isar0, aa64isar1, aa64mmfr2 }
+ AA64Reg {
+ aa64isar0,
+ aa64isar1,
+ #[cfg(test)]
+ #[cfg(not(portable_atomic_pre_llvm_18))]
+ aa64isar3,
+ #[cfg(test)]
+ #[cfg(portable_atomic_pre_llvm_18)]
+ aa64isar3: 0,
+ aa64mmfr2,
+ }
}
}
}
@@ -122,10 +182,10 @@ mod imp {
use super::AA64Reg;
- // core::ffi::c_* (except c_void) requires Rust 1.64, libc requires Rust 1.63
+ // libc requires Rust 1.63
#[allow(non_camel_case_types)]
pub(super) mod ffi {
- pub(crate) use super::super::c_types::{c_char, c_int, c_size_t, c_void};
+ pub(crate) use crate::utils::ffi::{CStr, c_char, c_int, c_size_t, c_void};
sys_struct!({
// Defined in machine/armreg.h.
@@ -173,24 +233,21 @@ mod imp {
});
}
- pub(super) unsafe fn sysctl_cpu_id(name: &[u8]) -> Option<AA64Reg> {
+ pub(super) fn sysctl_cpu_id(name: &ffi::CStr) -> Option<AA64Reg> {
const OUT_LEN: ffi::c_size_t =
mem::size_of::<ffi::aarch64_sysctl_cpu_id>() as ffi::c_size_t;
- debug_assert_eq!(name.last(), Some(&0), "{:?}", name);
- debug_assert_eq!(name.iter().filter(|&&v| v == 0).count(), 1, "{:?}", name);
-
// SAFETY: all fields of aarch64_sysctl_cpu_id are zero-able and we use
// the result when machdep.cpuN.cpu_id sysctl was successful.
let mut buf: ffi::aarch64_sysctl_cpu_id = unsafe { mem::zeroed() };
let mut out_len = OUT_LEN;
// SAFETY:
- // - the caller must guarantee that `name` is ` machdep.cpuN.cpu_id` in a C string.
+ // - `name` a valid C string.
// - `out_len` does not exceed the size of the value at `buf`.
// - `sysctlbyname` is thread-safe.
let res = unsafe {
ffi::sysctlbyname(
- name.as_ptr().cast::<ffi::c_char>(),
+ name.as_ptr(),
(&mut buf as *mut ffi::aarch64_sysctl_cpu_id).cast::<ffi::c_void>(),
&mut out_len,
ptr::null_mut(),
@@ -203,6 +260,8 @@ mod imp {
Some(AA64Reg {
aa64isar0: buf.ac_aa64isar0,
aa64isar1: buf.ac_aa64isar1,
+ #[cfg(test)]
+ aa64isar3: 0,
aa64mmfr2: buf.ac_aa64mmfr2,
})
}
@@ -211,13 +270,18 @@ mod imp {
// Get system registers for cpu0.
// If failed, returns default because machdep.cpuN.cpu_id sysctl is not available.
// machdep.cpuN.cpu_id sysctl was added in NetBSD 9.0 so it is not available on older versions.
- // SAFETY: we passed a valid name in a C string.
// It is ok to check only cpu0, even if there are more CPUs.
// https://github.com/NetBSD/src/commit/bd9707e06ea7d21b5c24df6dfc14cb37c2819416
// https://github.com/golang/sys/commit/ef9fd89ba245e184bdd308f7f2b4f3c551fa5b0f
- match unsafe { sysctl_cpu_id(b"machdep.cpu0.cpu_id\0") } {
+ match sysctl_cpu_id(c!("machdep.cpu0.cpu_id")) {
Some(cpu_id) => cpu_id,
- None => AA64Reg { aa64isar0: 0, aa64isar1: 0, aa64mmfr2: 0 },
+ None => AA64Reg {
+ aa64isar0: 0,
+ aa64isar1: 0,
+ #[cfg(test)]
+ aa64isar3: 0,
+ aa64mmfr2: 0,
+ },
}
}
}
@@ -231,10 +295,9 @@ mod imp {
use super::AA64Reg;
- // core::ffi::c_* (except c_void) requires Rust 1.64, libc requires Rust 1.63
- #[allow(non_camel_case_types)]
+ // libc requires Rust 1.63
pub(super) mod ffi {
- pub(crate) use super::super::c_types::{c_int, c_size_t, c_uint, c_void};
+ pub(crate) use crate::utils::ffi::{c_int, c_size_t, c_uint, c_void};
sys_const!({
// Defined in sys/sysctl.h.
@@ -277,7 +340,13 @@ mod imp {
let aa64isar0 = sysctl64(&[ffi::CTL_MACHDEP, ffi::CPU_ID_AA64ISAR0]).unwrap_or(0);
let aa64isar1 = sysctl64(&[ffi::CTL_MACHDEP, ffi::CPU_ID_AA64ISAR1]).unwrap_or(0);
let aa64mmfr2 = sysctl64(&[ffi::CTL_MACHDEP, ffi::CPU_ID_AA64MMFR2]).unwrap_or(0);
- AA64Reg { aa64isar0, aa64isar1, aa64mmfr2 }
+ AA64Reg {
+ aa64isar0,
+ aa64isar1,
+ #[cfg(test)]
+ aa64isar3: 0,
+ aa64mmfr2,
+ }
}
fn sysctl64(mib: &[ffi::c_int]) -> Option<u64> {
@@ -317,41 +386,22 @@ mod imp {
)]
#[cfg(test)]
mod tests {
- use std::{
- process::Command,
- string::{String, ToString},
- };
-
use super::*;
#[test]
+ #[cfg_attr(portable_atomic_test_detect_false, ignore = "detection disabled")]
fn test_aa64reg() {
- let AA64Reg { aa64isar0, aa64isar1, aa64mmfr2 } = imp::aa64reg();
- std::eprintln!("aa64isar0={}", aa64isar0);
- std::eprintln!("aa64isar1={}", aa64isar1);
- std::eprintln!("aa64mmfr2={}", aa64mmfr2);
- if cfg!(target_os = "openbsd") {
- let output = Command::new("sysctl").arg("machdep").output().unwrap();
- assert!(output.status.success());
- let stdout = String::from_utf8(output.stdout).unwrap();
- // OpenBSD 7.1+
- assert_eq!(
- stdout.lines().find_map(|s| s.strip_prefix("machdep.id_aa64isar0=")).unwrap_or("0"),
- aa64isar0.to_string(),
- );
- assert_eq!(
- stdout.lines().find_map(|s| s.strip_prefix("machdep.id_aa64isar1=")).unwrap_or("0"),
- aa64isar1.to_string(),
- );
- // OpenBSD 7.3+
- assert_eq!(
- stdout.lines().find_map(|s| s.strip_prefix("machdep.id_aa64mmfr2=")).unwrap_or("0"),
- aa64mmfr2.to_string(),
- );
- }
+ let AA64Reg { aa64isar0, aa64isar1, aa64isar3, aa64mmfr2 } = imp::aa64reg();
+ test_helper::eprintln_nocapture!(
+ "aa64isar0={},aa64isar1={},aa64isar3={},aa64mmfr2={}",
+ aa64isar0,
+ aa64isar1,
+ aa64isar3,
+ aa64mmfr2,
+ );
let atomic = extract(aa64isar0, 23, 20);
- if detect().test(CpuInfo::HAS_LSE) {
- if detect().test(CpuInfo::HAS_LSE128) {
+ if detect().lse() {
+ if detect().lse128() {
assert_eq!(atomic, 0b0011);
} else {
assert_eq!(atomic, 0b0010);
@@ -360,13 +410,27 @@ mod tests {
assert_eq!(atomic, 0b0000);
}
let lrcpc = extract(aa64isar1, 23, 20);
- if detect().test(CpuInfo::HAS_RCPC3) {
- assert_eq!(lrcpc, 0b0011);
+ if detect().rcpc() {
+ if detect().rcpc2() {
+ if detect().rcpc3() {
+ assert_eq!(lrcpc, 0b0011);
+ } else {
+ assert_eq!(lrcpc, 0b0010);
+ }
+ } else {
+ assert_eq!(lrcpc, 0b0001);
+ }
} else {
- assert!(lrcpc < 0b0011, "{}", lrcpc);
+ assert_eq!(lrcpc, 0b0000);
+ }
+ let lsfe = extract(aa64isar3, 19, 16);
+ if detect().lsfe() {
+ assert_eq!(lsfe, 0b0001);
+ } else {
+ assert_eq!(lsfe, 0b0000);
}
let at = extract(aa64mmfr2, 35, 32);
- if detect().test(CpuInfo::HAS_LSE2) {
+ if detect().lse2() {
assert_eq!(at, 0b0001);
} else {
assert_eq!(at, 0b0000);
@@ -377,20 +441,24 @@ mod tests {
#[cfg(target_os = "netbsd")]
#[test]
fn test_alternative() {
- use c_types::*;
- use imp::ffi;
#[cfg(not(portable_atomic_no_asm))]
use std::arch::asm;
use std::{mem, ptr, vec, vec::Vec};
+
use test_helper::sys;
+ use super::imp::ffi;
+ use crate::utils::{RegISize, RegSize, ffi::*};
+
// Call syscall using asm instead of libc.
// Note that NetBSD does not guarantee the stability of raw syscall as
// much as Linux does (It may actually be stable enough, though: https://lists.llvm.org/pipermail/llvm-dev/2019-June/133393.html).
//
// This is currently used only for testing.
- unsafe fn sysctl_cpu_id_no_libc(name: &[&[u8]]) -> Result<AA64Reg, c_int> {
- // https://github.com/golang/go/blob/4badad8d477ffd7a6b762c35bc69aed82faface7/src/syscall/asm_netbsd_arm64.s
+ fn sysctl_cpu_id_no_libc(name: &[&[u8]]) -> Result<AA64Reg, c_int> {
+ // Refs:
+ // - https://github.com/NetBSD/src/blob/c3bf19e1d461f8b4d8812b91b48116a1e45c9d04/lib/libc/arch/aarch64/SYS.h
+ // - https://github.com/golang/go/blob/go1.25.0/src/syscall/asm_netbsd_arm64.s
#[inline]
unsafe fn sysctl(
name: *const c_int,
@@ -400,35 +468,38 @@ mod tests {
new_p: *const c_void,
new_len: c_size_t,
) -> Result<c_int, c_int> {
+ let mut n = sys::SYS___sysctl as RegSize;
+ let arg1 = ptr_reg!(name);
+ let arg2 = name_len as RegSize;
+ let arg3 = ptr_reg!(old_p);
+ let arg4 = ptr_reg!(old_len_p);
+ let arg5 = ptr_reg!(new_p);
+ let arg6 = new_len as RegSize;
+ let r: RegISize;
// SAFETY: the caller must uphold the safety contract.
unsafe {
- let mut n = sys::SYS___sysctl as u64;
- let r: i64;
asm!(
- "svc 0",
+ "svc 0", // #SYS_syscall
"b.cc 2f",
"mov x17, x0",
"mov x0, #-1",
"2:",
inout("x17") n,
- inout("x0") ptr_reg!(name) => r,
- inout("x1") name_len as u64 => _,
- in("x2") ptr_reg!(old_p),
- in("x3") ptr_reg!(old_len_p),
- in("x4") ptr_reg!(new_p),
- in("x5") new_len as u64,
+ inout("x0") arg1 => r,
+ inout("x1") arg2 => _,
+ in("x2") arg3,
+ in("x3") arg4,
+ in("x4") arg5,
+ in("x5") arg6,
+ // Do not use `preserves_flags` because AArch64 NetBSD syscalls modify the condition flags.
options(nostack),
);
- #[allow(clippy::cast_possible_truncation)]
- if r as c_int == -1 {
- Err(n as c_int)
- } else {
- Ok(r as c_int)
- }
}
+ #[allow(clippy::cast_possible_truncation)]
+ if r as c_int == -1 { Err(n as c_int) } else { Ok(r as c_int) }
}
- // https://github.com/golang/sys/blob/4badad8d477ffd7a6b762c35bc69aed82faface7/cpu/cpu_netbsd_arm64.go.
+ // https://github.com/golang/sys/blob/v0.35.0/cpu/cpu_netbsd_arm64.go
fn sysctl_nodes(mib: &mut Vec<i32>) -> Result<Vec<sys::sysctlnode>, i32> {
mib.push(sys::CTL_QUERY);
let mut q_node = sys::sysctlnode {
@@ -500,15 +571,48 @@ mod tests {
Ok(AA64Reg {
aa64isar0: buf.ac_aa64isar0,
aa64isar1: buf.ac_aa64isar1,
+ aa64isar3: 0,
aa64mmfr2: buf.ac_aa64mmfr2,
})
}
- unsafe {
- assert_eq!(
- imp::sysctl_cpu_id(b"machdep.cpu0.cpu_id\0").unwrap(),
- sysctl_cpu_id_no_libc(&[b"machdep", b"cpu0", b"cpu_id"]).unwrap()
- );
+ assert_eq!(
+ imp::sysctl_cpu_id(c!("machdep.cpu0.cpu_id")).unwrap(),
+ sysctl_cpu_id_no_libc(&[b"machdep", b"cpu0", b"cpu_id"]).unwrap()
+ );
+ }
+ #[cfg(target_os = "openbsd")]
+ #[test]
+ fn test_alternative() {
+ use std::{format, process::Command, string::String};
+
+ // Call sysctl command instead of libc API.
+ //
+ // This is used only for testing.
+ struct SysctlMachdepOutput(String);
+ impl SysctlMachdepOutput {
+ fn new() -> Self {
+ let output = Command::new("sysctl").arg("machdep").output().unwrap();
+ assert!(output.status.success());
+ let stdout = String::from_utf8(output.stdout).unwrap();
+ Self(stdout)
+ }
+ fn field(&self, name: &str) -> Option<u64> {
+ Some(
+ self.0
+ .lines()
+ .find_map(|s| s.strip_prefix(&format!("{}=", name)))?
+ .parse()
+ .unwrap(),
+ )
+ }
}
+
+ let AA64Reg { aa64isar0, aa64isar1, aa64isar3, aa64mmfr2 } = imp::aa64reg();
+ let sysctl_output = SysctlMachdepOutput::new();
+ assert_eq!(aa64isar0, sysctl_output.field("machdep.id_aa64isar0").unwrap_or(0));
+ assert_eq!(aa64isar1, sysctl_output.field("machdep.id_aa64isar1").unwrap_or(0));
+ assert_eq!(aa64isar3, sysctl_output.field("machdep.id_aa64isar3").unwrap_or(0));
+ assert_eq!(aa64mmfr2, sysctl_output.field("machdep.id_aa64mmfr2").unwrap_or(0));
}
}
### external/vendor/portable-atomic/src/imp/detect/aarch64_apple.rs
@@ -3,12 +3,25 @@
/*
Run-time CPU feature detection on AArch64 Apple targets by using sysctlbyname.
-On macOS, this module is currently only enabled on tests because AArch64 macOS
-always supports FEAT_LSE and FEAT_LSE2 (see build script for more).
+On macOS, this module is currently only enabled on tests because there are no
+instructions that were not available on the M1 but are now available on the
+latest Apple hardware and this library currently wants to use:
-If macOS supporting FEAT_LSE128/FEAT_LRCPC3 becomes popular in the future, this module will
-be used to support outline-atomics for FEAT_LSE128/FEAT_LRCPC3.
-M4 is Armv9.2 and it doesn't support FEAT_LSE128/FEAT_LRCPC3.
+```console
+$ LC_ALL=C comm -23 <(rustc --print cfg --target aarch64-apple-darwin -C target-cpu=apple-m4 | grep -F target_feature) <(rustc --print cfg --target aarch64-apple-darwin | grep -F target_feature)
+target_feature="bf16"
+target_feature="bti"
+target_feature="ecv"
+target_feature="i8mm"
+target_feature="sme"
+target_feature="sme-f64f64"
+target_feature="sme-i16i64"
+target_feature="sme2"
+target_feature="v8.5a"
+target_feature="v8.6a"
+target_feature="v8.7a"
+target_feature="wfxt"
+```
Refs: https://developer.apple.com/documentation/kernel/1387446-sysctlbyname/determining_instruction_set_characteristics
@@ -22,10 +35,9 @@ include!("common.rs");
use core::{mem, ptr};
-// core::ffi::c_* (except c_void) requires Rust 1.64, libc requires Rust 1.63
-#[allow(non_camel_case_types)]
+// libc requires Rust 1.63
mod ffi {
- pub(crate) use super::c_types::{c_char, c_int, c_size_t, c_void};
+ pub(crate) use crate::utils::ffi::{CStr, c_char, c_int, c_size_t, c_void};
sys_fn!({
extern "C" {
@@ -42,21 +54,18 @@ mod ffi {
});
}
-unsafe fn sysctlbyname32(name: &[u8]) -> Option<u32> {
+fn sysctlbyname32(name: &ffi::CStr) -> Option<u32> {
const OUT_LEN: ffi::c_size_t = mem::size_of::<u32>() as ffi::c_size_t;
- debug_assert_eq!(name.last(), Some(&0), "{:?}", name);
- debug_assert_eq!(name.iter().filter(|&&v| v == 0).count(), 1, "{:?}", name);
-
let mut out = 0_u32;
let mut out_len = OUT_LEN;
// SAFETY:
- // - the caller must guarantee that `name` a valid C string.
+ // - `name` a valid C string.
// - `out_len` does not exceed the size of `out`.
// - `sysctlbyname` is thread-safe.
let res = unsafe {
ffi::sysctlbyname(
- name.as_ptr().cast::<ffi::c_char>(),
+ name.as_ptr(),
(&mut out as *mut u32).cast::<ffi::c_void>(),
&mut out_len,
ptr::null_mut(),
@@ -72,30 +81,30 @@ unsafe fn sysctlbyname32(name: &[u8]) -> Option<u32> {
#[cold]
fn _detect(info: &mut CpuInfo) {
- // hw.optional.armv8_1_atomics is available on macOS 11+ (note: AArch64 support was added in macOS 11),
+ macro_rules! check {
+ ($flag:ident, $($name:tt) ||+) => {
+ if $(sysctlbyname32(c!($name)).unwrap_or(0) != 0) ||+ {
+ info.set(CpuInfoFlag::$flag);
+ }
+ };
+ }
+
+ // On macOS, AArch64 support was added in macOS 11,
+ // hw.optional.armv8_1_atomics is available on macOS 11+,
// hw.optional.arm.FEAT_* are only available on macOS 12+.
// Query both names in case future versions of macOS remove the old name.
// https://github.com/golang/go/commit/c15593197453b8bf90fc3a9080ba2afeaf7934ea
// https://github.com/google/boringssl/commit/91e0b11eba517d83b910b20fe3740eeb39ecb37e
- // SAFETY: we passed a valid C string.
- if unsafe {
- sysctlbyname32(b"hw.optional.arm.FEAT_LSE\0").unwrap_or(0) != 0
- || sysctlbyname32(b"hw.optional.armv8_1_atomics\0").unwrap_or(0) != 0
- } {
- info.set(CpuInfo::HAS_LSE);
- }
- // SAFETY: we passed a valid C string.
- if unsafe { sysctlbyname32(b"hw.optional.arm.FEAT_LSE2\0").unwrap_or(0) != 0 } {
- info.set(CpuInfo::HAS_LSE2);
- }
- // SAFETY: we passed a valid C string.
- if unsafe { sysctlbyname32(b"hw.optional.arm.FEAT_LSE128\0").unwrap_or(0) != 0 } {
- info.set(CpuInfo::HAS_LSE128);
- }
- // SAFETY: we passed a valid C string.
- if unsafe { sysctlbyname32(b"hw.optional.arm.FEAT_LRCPC3\0").unwrap_or(0) != 0 } {
- info.set(CpuInfo::HAS_RCPC3);
- }
+ check!(lse, "hw.optional.arm.FEAT_LSE" || "hw.optional.armv8_1_atomics");
+ check!(lse2, "hw.optional.arm.FEAT_LSE2");
+ check!(lse128, "hw.optional.arm.FEAT_LSE128");
+ #[cfg(test)]
+ check!(lsfe, "hw.optional.arm.FEAT_LSFE");
+ #[cfg(test)]
+ check!(rcpc, "hw.optional.arm.FEAT_LRCPC");
+ #[cfg(test)]
+ check!(rcpc2, "hw.optional.arm.FEAT_LRCPC2");
+ check!(rcpc3, "hw.optional.arm.FEAT_LRCPC3");
}
#[allow(
@@ -107,38 +116,29 @@ fn _detect(info: &mut CpuInfo) {
)]
#[cfg(test)]
mod tests {
- use super::*;
+ use std::{format, process::Command, str, string::String};
- #[test]
- fn test_macos() {
- unsafe {
- assert_eq!(sysctlbyname32(b"hw.optional.armv8_1_atomics\0"), Some(1));
- assert_eq!(sysctlbyname32(b"hw.optional.arm.FEAT_LSE\0"), Some(1));
- assert_eq!(sysctlbyname32(b"hw.optional.arm.FEAT_LSE2\0"), Some(1));
- assert_eq!(sysctlbyname32(b"hw.optional.arm.FEAT_LSE128\0"), None);
- assert_eq!(std::io::Error::last_os_error().kind(), std::io::ErrorKind::NotFound);
- assert_eq!(sysctlbyname32(b"hw.optional.arm.FEAT_LRCPC\0"), Some(1));
- assert_eq!(sysctlbyname32(b"hw.optional.arm.FEAT_LRCPC2\0"), Some(1));
- assert_eq!(sysctlbyname32(b"hw.optional.arm.FEAT_LRCPC3\0"), None);
- assert_eq!(std::io::Error::last_os_error().kind(), std::io::ErrorKind::NotFound);
- }
- }
+ use super::*;
- #[cfg(target_pointer_width = "64")]
#[test]
fn test_alternative() {
- use c_types::*;
- #[cfg(not(portable_atomic_no_asm))]
- use std::arch::asm;
- use std::mem;
- use test_helper::sys;
+ use crate::utils::ffi::*;
+
// Call syscall using asm instead of libc.
// Note that macOS does not guarantee the stability of raw syscall.
// (And they actually changed it: https://go-review.googlesource.com/c/go/+/25495)
//
// This is currently used only for testing.
- unsafe fn sysctlbyname32_no_libc(name: &[u8]) -> Result<u32, c_int> {
- // https://github.com/apple-oss-distributions/xnu/blob/8d741a5de7ff4191bf97d57b9f54c2f6d4a15585/bsd/kern/syscalls.master#L298
+ fn sysctlbyname32_no_libc(name: &CStr) -> Result<u32, c_int> {
+ #[cfg(not(portable_atomic_no_asm))]
+ use std::arch::asm;
+ use std::mem;
+
+ use test_helper::sys;
+
+ use crate::utils::{RegISize, RegSize};
+
+ // Refs: https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/libsyscall/custom/SYS.h#L427
#[inline]
unsafe fn sysctl(
name: *const c_int,
@@ -157,37 +157,40 @@ mod tests {
const SYSCALL_NUMBER_MASK: u64 = !SYSCALL_CLASS_MASK;
(SYSCALL_CLASS_UNIX << SYSCALL_CLASS_SHIFT) | (SYSCALL_NUMBER_MASK & n)
}
- #[allow(clippy::cast_possible_truncation)]
+ // https://github.com/apple-oss-distributions/xnu/blob/8d741a5de7ff4191bf97d57b9f54c2f6d4a15585/bsd/kern/syscalls.master#L298
+ let mut n = syscall_construct_unix(202);
+ let arg1 = ptr_reg!(name);
+ let arg2 = name_len as RegSize;
+ let arg3 = ptr_reg!(old_p);
+ let arg4 = ptr_reg!(old_len_p);
+ let arg5 = ptr_reg!(new_p);
+ let arg6 = new_len as RegSize;
+ let r: RegISize;
// SAFETY: the caller must uphold the safety contract.
unsafe {
- // https://github.com/apple-oss-distributions/xnu/blob/8d741a5de7ff4191bf97d57b9f54c2f6d4a15585/bsd/kern/syscalls.master#L4
- let mut n = syscall_construct_unix(202);
- let r: i64;
asm!(
- "svc 0",
+ "svc 0x80", // #SWI_SYSCALL https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/osfmk/mach/arm/vm_param.h#L417
"b.cc 2f",
"mov x16, x0",
"mov x0, #-1",
"2:",
inout("x16") n,
- inout("x0") ptr_reg!(name) => r,
- inout("x1") name_len as u64 => _,
- in("x2") ptr_reg!(old_p),
- in("x3") ptr_reg!(old_len_p),
- in("x4") ptr_reg!(new_p),
- in("x5") new_len as u64,
+ inout("x0") arg1 => r,
+ inout("x1") arg2 => _,
+ in("x2") arg3,
+ in("x3") arg4,
+ in("x4") arg5,
+ in("x5") arg6,
+ // Do not use `preserves_flags` because AArch64 Darwin syscalls modify the condition flags.
options(nostack),
);
- if r as c_int == -1 {
- Err(n as c_int)
- } else {
- Ok(r as c_int)
- }
}
+ #[allow(clippy::cast_possible_truncation)]
+ if r as c_int == -1 { Err(n as c_int) } else { Ok(r as c_int) }
}
// https://github.com/apple-oss-distributions/Libc/blob/af11da5ca9d527ea2f48bb7efbd0f0f2a4ea4812/gen/FreeBSD/sysctlbyname.c
unsafe fn sysctlbyname(
- name: &[u8],
+ name: &CStr,
old_p: *mut c_void,
old_len_p: *mut c_size_t,
new_p: *mut c_void,
@@ -207,9 +210,9 @@ mod tests {
real_oid.as_mut_ptr().cast::<c_void>(),
&mut oid_len,
name.as_ptr().cast::<c_void>() as *mut c_void,
- name.len() - 1,
- )?
- };
+ name.to_bytes_with_nul().len() - 1,
+ )?;
+ }
oid_len /= mem::size_of::<c_int>();
#[allow(clippy::cast_possible_truncation)]
unsafe {
@@ -219,13 +222,9 @@ mod tests {
const OUT_LEN: ffi::c_size_t = mem::size_of::<u32>() as ffi::c_size_t;
- debug_assert_eq!(name.last(), Some(&0), "{:?}", name);
- debug_assert_eq!(name.iter().filter(|&&v| v == 0).count(), 1, "{:?}", name);
-
let mut out = 0_u32;
let mut out_len = OUT_LEN;
// SAFETY:
- // - the caller must guarantee that `name` a valid C string.
// - `out_len` does not exceed the size of `out`.
// - `sysctlbyname` is thread-safe.
let res = unsafe {
@@ -242,21 +241,60 @@ mod tests {
Ok(out)
}
- for name in [
- &b"hw.optional.armv8_1_atomics\0"[..],
- b"hw.optional.arm.FEAT_LSE\0",
- b"hw.optional.arm.FEAT_LSE2\0",
- b"hw.optional.arm.FEAT_LSE128\0",
- b"hw.optional.arm.FEAT_LRCPC\0",
- b"hw.optional.arm.FEAT_LRCPC2\0",
- b"hw.optional.arm.FEAT_LRCPC3\0",
+ // Call sysctl command instead of libc API.
+ //
+ // This is used only for testing.
+ struct SysctlHwOptionalOutput(String);
+ impl SysctlHwOptionalOutput {
+ fn new() -> Self {
+ let output = Command::new("sysctl").arg("hw.optional").output().unwrap();
+ assert!(output.status.success());
+ let stdout = String::from_utf8(output.stdout).unwrap();
+ test_helper::eprintln_nocapture!("sysctl hw.optional:\n{}", stdout);
+ Self(stdout)
+ }
+ fn field(&self, name: &CStr) -> Option<u32> {
+ let name = name.to_bytes_with_nul();
+ let name = str::from_utf8(&name[..name.len() - 1]).unwrap();
+ Some(
+ self.0
+ .lines()
+ .find_map(|s| s.strip_prefix(&format!("{}: ", name)))?
+ .parse()
+ .unwrap(),
+ )
+ }
+ }
+
+ let sysctl_output = SysctlHwOptionalOutput::new();
+ for (name, expected_on_macos) in [
+ (c!("hw.optional.arm.FEAT_LSE"), Some(1)),
+ (c!("hw.optional.armv8_1_atomics"), Some(1)),
+ (c!("hw.optional.arm.FEAT_LSE2"), Some(1)),
+ (c!("hw.optional.arm.FEAT_LSE128"), None),
+ (c!("hw.optional.arm.FEAT_LSFE"), None),
+ (c!("hw.optional.arm.FEAT_LRCPC"), Some(1)),
+ (c!("hw.optional.arm.FEAT_LRCPC2"), Some(1)),
+ (c!("hw.optional.arm.FEAT_LRCPC3"), None),
] {
- unsafe {
- if let Some(res) = sysctlbyname32(name) {
- assert_eq!(res, sysctlbyname32_no_libc(name).unwrap());
- } else {
- assert_eq!(sysctlbyname32_no_libc(name).unwrap_err(), libc::ENOENT);
- }
+ let res = sysctlbyname32(name);
+ if res.is_none() {
+ assert_eq!(std::io::Error::last_os_error().kind(), std::io::ErrorKind::NotFound);
+ }
+ if cfg!(any(target_os = "macos", target_abi = "macabi")) {
+ assert_eq!(
+ res,
+ expected_on_macos,
+ "{}",
+ str::from_utf8(name.to_bytes_with_nul()).unwrap()
+ );
+ }
+ if let Some(res) = res {
+ assert_eq!(res, sysctlbyname32_no_libc(name).unwrap());
+ assert_eq!(res, sysctl_output.field(name).unwrap());
+ } else {
+ assert_eq!(sysctlbyname32_no_libc(name).unwrap_err(), libc::ENOENT);
+ assert!(sysctl_output.field(name).is_none());
}
}
}
### external/vendor/portable-atomic/src/imp/detect/aarch64_fuchsia.rs
@@ -15,8 +15,10 @@ include!("common.rs");
#[allow(non_camel_case_types)]
mod ffi {
- // https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/zircon/system/public/zircon/types.h
- pub(crate) type zx_status_t = i32;
+ sys_type!({
+ // https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/zircon/system/public/zircon/types.h
+ pub(crate) type zx_status_t = i32;
+ });
sys_const!({
// https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/zircon/system/public/zircon/errors.h
@@ -48,9 +50,14 @@ fn zx_system_get_features(kind: u32) -> u32 {
#[cold]
fn _detect(info: &mut CpuInfo) {
let features = zx_system_get_features(ffi::ZX_FEATURE_KIND_CPU);
- if features & ffi::ZX_ARM64_FEATURE_ISA_ATOMICS != 0 {
- info.set(CpuInfo::HAS_LSE);
+ macro_rules! check {
+ ($flag:ident, $bit:ident) => {
+ if features & ffi::$bit != 0 {
+ info.set(CpuInfoFlag::$flag);
+ }
+ };
}
+ check!(lse, ZX_ARM64_FEATURE_ISA_ATOMICS);
}
#[allow(
@@ -67,24 +74,10 @@ mod tests {
#[test]
fn test_fuchsia() {
let features = zx_system_get_features(ffi::ZX_FEATURE_KIND_CPU);
+ test_helper::eprintln_nocapture!(
+ "zx_system_get_features(ZX_FEATURE_KIND_CPU): {:b}",
+ features
+ );
assert_ne!(features, 0);
- std::eprintln!("features: {:b}", features);
}
-
- // Static assertions for FFI bindings.
- // This checks that FFI bindings defined in this crate and FFI bindings
- // generated for the platform's latest header file using bindgen have
- // compatible signatures.
- // Since this is static assertion, we can detect problems with
- // `cargo check --tests --target <target>` run in CI (via TESTS=1 build.sh)
- // without actually running tests on these platforms.
- // As for constants, they are checked by static assertions generated by sys_const!.
- // See also https://github.com/taiki-e/test-helper/blob/HEAD/tools/codegen/src/ffi.rs.
- // TODO(codegen): auto-generate this test
- #[allow(clippy::cast_possible_wrap, clippy::cast_sign_loss, clippy::cast_possible_truncation)]
- const _: fn() = || {
- use test_helper::sys;
- // TODO(codegen): zx_system_get_features
- let _: ffi::zx_status_t = 0 as sys::zx_status_t;
- };
}
### external/vendor/portable-atomic/src/imp/detect/aarch64_illumos.rs
@@ -6,20 +6,24 @@ Run-time CPU feature detection on AArch64 illumos by using getisax.
As of nightly-2024-09-07, is_aarch64_feature_detected doesn't support run-time detection on illumos.
https://github.com/rust-lang/stdarch/blob/d9466edb4c53cece8686ee6e17b028436ddf4151/crates/std_detect/src/detect/mod.rs
-Run-time detection on AArch64 illumos is currently disabled by default as AArch64 port is experimental.
+Run-time detection on AArch64 illumos is currently disabled by default as experimental
+because illumos AArch64 port is experimental and we cannot run tests on the VM or real machine.
*/
include!("common.rs");
-// core::ffi::c_* (except c_void) requires Rust 1.64, libc requires Rust 1.63
-#[allow(non_camel_case_types)]
+// libc requires Rust 1.63
mod ffi {
- pub(crate) use super::c_types::c_uint;
+ pub(crate) use crate::utils::ffi::c_uint;
sys_const!({
// Defined in sys/auxv_aarch64.h.
// https://github.com/richlowe/illumos-gate/blob/arm64-gate/usr/src/uts/common/sys/auxv_aarch64.h
pub(crate) const AV_AARCH64_LSE: u32 = 1 << 15;
+ #[cfg(test)]
+ pub(crate) const AV_AARCH64_LRCPC: u32 = 1 << 28;
+ #[cfg(test)]
+ pub(crate) const AV_AARCH64_2_ILRCPC: u32 = 1 << 1;
pub(crate) const AV_AARCH64_2_LSE2: u32 = 1 << 2;
});
@@ -41,10 +45,19 @@ fn _detect(info: &mut CpuInfo) {
unsafe {
ffi::getisax(out.as_mut_ptr(), OUT_LEN);
}
- if out[0] & ffi::AV_AARCH64_LSE != 0 {
- info.set(CpuInfo::HAS_LSE);
- }
- if out[1] & ffi::AV_AARCH64_2_LSE2 != 0 {
- info.set(CpuInfo::HAS_LSE2);
+ macro_rules! check {
+ ($x:ident, $flag:ident, $bit:ident) => {
+ if $x & ffi::$bit != 0 {
+ info.set(CpuInfoFlag::$flag);
+ }
+ };
}
+ let v1 = out[0];
+ check!(v1, lse, AV_AARCH64_LSE);
+ #[cfg(test)]
+ check!(v1, rcpc, AV_AARCH64_LRCPC);
+ let v2 = out[1];
+ #[cfg(test)]
+ check!(v2, rcpc2, AV_AARCH64_2_ILRCPC);
+ check!(v2, lse2, AV_AARCH64_2_LSE2);
}
### external/vendor/portable-atomic/src/imp/detect/aarch64_windows.rs
@@ -11,68 +11,47 @@ Refs: https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-p
include!("common.rs");
-// windows-sys requires Rust 1.60
-#[allow(clippy::upper_case_acronyms)]
+// windows-sys requires Rust 1.71
+#[allow(non_camel_case_types, clippy::upper_case_acronyms)]
mod ffi {
- pub(crate) type DWORD = u32;
- pub(crate) type BOOL = i32;
-
- pub(crate) const FALSE: BOOL = 0;
-
- // Defined in winnt.h of Windows SDK.
- pub(crate) const PF_ARM_V81_ATOMIC_INSTRUCTIONS_AVAILABLE: DWORD = 34;
-
- extern "system" {
- // https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-isprocessorfeaturepresent
- pub(crate) fn IsProcessorFeaturePresent(ProcessorFeature: DWORD) -> BOOL;
- }
+ sys_type!({
+ pub(crate) type [Win32::System::Threading] PROCESSOR_FEATURE_ID = u32;
+ pub(crate) type [core] BOOL = i32;
+ });
+
+ sys_const!({
+ pub(crate) const [Win32::Foundation] FALSE: BOOL = 0;
+
+ // Defined in winnt.h of Windows SDK.
+ pub(crate) const [Win32::System::Threading]
+ PF_ARM_V81_ATOMIC_INSTRUCTIONS_AVAILABLE: PROCESSOR_FEATURE_ID = 34;
+ #[cfg(test)]
+ pub(crate) const [Win32::System::Threading]
+ PF_ARM_V83_LRCPC_INSTRUCTIONS_AVAILABLE: PROCESSOR_FEATURE_ID = 45;
+ });
+
+ sys_fn!({
+ extern "system" {
+ // https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-isprocessorfeaturepresent
+ pub(crate) fn [Win32::System::Threading] IsProcessorFeaturePresent(
+ ProcessorFeature: PROCESSOR_FEATURE_ID,
+ ) -> BOOL;
+ }
+ });
}
#[cold]
fn _detect(info: &mut CpuInfo) {
- // SAFETY: calling IsProcessorFeaturePresent is safe, and FALSE is also
- // returned if the HAL does not support detection of the specified feature.
- if unsafe {
- ffi::IsProcessorFeaturePresent(ffi::PF_ARM_V81_ATOMIC_INSTRUCTIONS_AVAILABLE) != ffi::FALSE
- } {
- info.set(CpuInfo::HAS_LSE);
+ macro_rules! check {
+ ($flag:ident, $bit:ident) => {
+ // SAFETY: calling IsProcessorFeaturePresent is safe, and FALSE is also
+ // returned if the HAL does not support detection of the specified feature.
+ if unsafe { ffi::IsProcessorFeaturePresent(ffi::$bit) != ffi::FALSE } {
+ info.set(CpuInfoFlag::$flag);
+ }
+ };
}
-}
-
-#[allow(
- clippy::alloc_instead_of_core,
- clippy::std_instead_of_alloc,
- clippy::std_instead_of_core,
- clippy::undocumented_unsafe_blocks,
- clippy::wildcard_imports
-)]
-#[cfg(test)]
-mod tests {
- use super::*;
-
- // Static assertions for FFI bindings.
- // This checks that FFI bindings defined in this crate and FFI bindings defined
- // in windows-sys have compatible signatures (or the same values if constants).
- // Since this is static assertion, we can detect problems with
- // `cargo check --tests --target <target>` run in CI (via TESTS=1 build.sh)
- // without actually running tests on these platforms.
- // (Unlike libc, windows-sys programmatically generates bindings from Windows
- // API metadata, so it should be enough to check compatibility with the
- // windows-sys' signatures/values.)
- // See also https://github.com/taiki-e/test-helper/blob/HEAD/tools/codegen/src/ffi.rs.
- // TODO(codegen): auto-generate this test
- #[allow(clippy::cast_possible_wrap, clippy::cast_sign_loss, clippy::cast_possible_truncation)]
- const _: fn() = || {
- let _: ffi::DWORD = 0 as windows_sys::Win32::System::Threading::PROCESSOR_FEATURE_ID;
- let _: ffi::BOOL = 0 as windows_sys::Win32::Foundation::BOOL;
- let mut _is_processor_feature_present: unsafe extern "system" fn(ffi::DWORD) -> ffi::BOOL =
- ffi::IsProcessorFeaturePresent;
- _is_processor_feature_present =
- windows_sys::Win32::System::Threading::IsProcessorFeaturePresent;
- static_assert!(ffi::FALSE == windows_sys::Win32::Foundation::FALSE);
- static_assert!(
- ffi::PF_ARM_V81_ATOMIC_INSTRUCTIONS_AVAILABLE
- == windows_sys::Win32::System::Threading::PF_ARM_V81_ATOMIC_INSTRUCTIONS_AVAILABLE
- );
- };
+ check!(lse, PF_ARM_V81_ATOMIC_INSTRUCTIONS_AVAILABLE);
+ #[cfg(test)]
+ check!(rcpc, PF_ARM_V83_LRCPC_INSTRUCTIONS_AVAILABLE);
}
### external/vendor/portable-atomic/src/imp/detect/auxv.rs
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
/*
-Run-time CPU feature detection on AArch64/PowerPC64 Linux/Android/FreeBSD/OpenBSD by parsing ELF auxiliary vectors.
+Run-time CPU feature detection on AArch64/Arm/PowerPC64 Linux/Android/FreeBSD/OpenBSD by parsing ELF auxiliary vectors.
Supported platforms:
- Linux 6.4+ (through prctl)
@@ -14,45 +14,71 @@ Supported platforms:
- aarch64 (glibc 2.17+ https://github.com/bminor/glibc/blob/glibc-2.17/NEWS#L36)
- powerpc64 (le) (glibc 2.19+ or RHEL/CentOS's patched glibc 2.17+ https://github.com/bminor/glibc/blob/glibc-2.19/NEWS#L108)
Not always available on:
+ - arm (glibc 2.1+ https://github.com/bminor/glibc/blob/glibc-2.1/NEWS#L97)
- powerpc64 (be) (glibc 2.3+ https://github.com/bminor/glibc/blob/glibc-2.3/NEWS#L56)
- Since Rust 1.64, std requires glibc 2.17+ https://blog.rust-lang.org/2022/08/01/Increasing-glibc-kernel-requirements.html
+ Since Rust 1.64, std requires glibc 2.17+ https://blog.rust-lang.org/2022/08/01/Increasing-glibc-kernel-requirements
- musl 1.1.0+ (through getauxval)
https://github.com/bminor/musl/commit/21ada94c4b8c01589367cea300916d7db8461ae7
Always available on:
- aarch64 (musl 1.1.7+ https://github.com/bminor/musl/blob/v1.1.7/WHATSNEW#L1422)
- powerpc64 (musl 1.1.15+ https://github.com/bminor/musl/blob/v1.1.15/WHATSNEW#L1702)
+ Not always available on:
+ - arm (musl 0.8.3+ https://github.com/bminor/musl/blob/v0.8.3/WHATSNEW#L354)
+ At least since Rust 1.15, std requires musl 1.1.14+ https://github.com/rust-lang/rust/blob/1.15.0/src/ci/docker/x86_64-musl/build-musl.sh#L15
+ Since Rust 1.18, std requires musl 1.1.16+ https://github.com/rust-lang/rust/pull/41089
+ Since Rust 1.23, std requires musl 1.1.17+ https://github.com/rust-lang/rust/pull/45393
+ Since Rust 1.25, std requires musl 1.1.18+ https://github.com/rust-lang/rust/pull/47283
+ Since Rust 1.29, std requires musl 1.1.19+ https://github.com/rust-lang/rust/pull/52087
Since Rust 1.31, std requires musl 1.1.20+ https://github.com/rust-lang/rust/pull/54430
Since Rust 1.37, std requires musl 1.1.22+ https://github.com/rust-lang/rust/pull/61252
Since Rust 1.46, std requires musl 1.1.24+ https://github.com/rust-lang/rust/pull/73089
- Since Rust 1.71, std requires musl 1.2.3+ https://blog.rust-lang.org/2023/05/09/Updating-musl-targets.html
+ Since Rust 1.71, std requires musl 1.2.3+ https://blog.rust-lang.org/2023/05/09/Updating-musl-targets
+ OpenHarmony uses a fork of musl 1.2 https://gitee.com/openharmony/docs/blob/master/en/application-dev/reference/native-lib/musl.md
- uClibc-ng 1.0.43+ (through getauxval)
https://github.com/wbx-github/uclibc-ng/commit/d869bb1600942c01a77539128f9ba5b5b55ad647
Not always available on:
- aarch64 (uClibc-ng 1.0.22+ https://github.com/wbx-github/uclibc-ng/commit/dba942c80dc2cfa5768a856fff98e22a755fdd27)
+ - arm (uClibc-ng 1.0.0+ https://github.com/wbx-github/uclibc-ng/tree/v1.0.0/libc/sysdeps/linux)
(powerpc64 is not supported https://github.com/wbx-github/uclibc-ng/commit/d4d4f37fda7fa57e57132ff2f0d735ce7cc2178e)
- Picolibc 1.4.6+ (through getauxval)
https://github.com/picolibc/picolibc/commit/19bfe51d62ad7e32533c7f664b5bca8e26286e31
+ The implementation always return 0 (as of 1.8.10): https://github.com/picolibc/picolibc/blob/1.8.10/newlib/libc/picolib/getauxval.c
- Android 4.3+ (API level 18+) (through getauxval)
https://github.com/aosp-mirror/platform_bionic/commit/2c5153b043b44e9935a334ae9b2d5a4bc5258b40
https://github.com/aosp-mirror/platform_bionic/commit/655e430b28d7404f763e7ebefe84fba5a387666d
Always available on:
- 64-bit architectures (Android 5.0+ (API level 21+) https://android-developers.googleblog.com/2014/10/whats-new-in-android-50-lollipop.html)
- Since Rust 1.68, std requires API level 19+ https://blog.rust-lang.org/2023/01/09/android-ndk-update-r25.html
+ Not always available on:
+ - arm
+ Since Rust 1.68, std requires API level 19+ https://blog.rust-lang.org/2023/01/09/android-ndk-update-r25
Since Rust 1.82, std requires API level 21+ https://github.com/rust-lang/rust/pull/120593
- FreeBSD 12.0+ and 11.4+ (through elf_aux_info)
https://github.com/freebsd/freebsd-src/commit/0b08ae2120cdd08c20a2b806e2fcef4d0a36c470
https://github.com/freebsd/freebsd-src/blob/release/11.4.0/sys/sys/auxv.h
+ Always available on:
+ - arm (v7) (FreeBSD 12.0+ https://www.freebsd.org/releases/12.0R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
+ - powerpc64 (le) (FreeBSD 12.4+ https://www.freebsd.org/releases/12.4R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
Not always available on:
- - aarch64 (FreeBSD 11.0+ https://www.freebsd.org/releases/11.0R/announce)
- - powerpc64 (FreeBSD 9.0+ https://www.freebsd.org/releases/9.0R/announce)
+ - aarch64 (FreeBSD 11.0+ https://www.freebsd.org/releases/11.0R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
+ - arm (v6) (FreeBSD 10.1+ https://www.freebsd.org/releases/10.1R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
+ - powerpc64 (be) (FreeBSD 9.0+ https://www.freebsd.org/releases/9.0R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
Since Rust 1.75, std requires FreeBSD 12+ https://github.com/rust-lang/rust/pull/114521
- Since Rust 1.84, std requires FreeBSD 13+ https://github.com/rust-lang/rust/pull/120869
+ Dropping support for FreeBSD 12 in std was decided in https://github.com/rust-lang/rust/pull/120869,
+ but the actual update to the FreeBSD 13 toolchain was attempted twice, but both times there were
+ problems, so they were reverted: https://github.com/rust-lang/rust/pull/132228 https://github.com/rust-lang/rust/pull/136582
- OpenBSD 7.6+ (through elf_aux_info)
https://github.com/openbsd/src/commit/ef873df06dac50249b2dd380dc6100eee3b0d23d
Not always available on:
- aarch64 (OpenBSD 6.1+ https://www.openbsd.org/61.html)
+ - arm (OpenBSD 6.0+ https://www.openbsd.org/60.html)
- powerpc64 (OpenBSD 6.8+ https://www.openbsd.org/68.html)
+On L4Re which uses uClibc-ng, a dummy getauxval was added first in 2020 that always returns 0 (https://github.com/kernkonzept/l4re-core/commit/e88fa67198074d3e6b4983c5c8af1538e2089ff3),
+then implemented in 2024 (https://github.com/kernkonzept/l4re-core/commit/3ee2a50dd1b3bc22955e593004990887a0a5b4a3).
+However, getauxval(AT_HWCAP*) always returns 0 (as of 2025-12-25). (see tests/l4re test)
+On Redox, getauxval is available since 0.5.0 (https://github.com/redox-os/relibc/commit/f9f752d74c4f1f56a89c0fcdd5cab63d2380fe09),
+but the implementation always return 0 (as of 2025-12-25). https://github.com/redox-os/relibc/blob/bb3cadfca4f7e885e600eba1276a9d24bbddb531/src/header/sys_auxv/mod.rs
+
On platforms that we can assume that getauxval/elf_aux_info is always available, we directly call
them on except for musl with static linking. (At this time, we also retain compatibility with
versions that reached EoL or no longer supported by `std`, with the exception of AArch64 FreeBSD described below.)
@@ -62,6 +88,14 @@ requirements: https://github.com/rust-lang/rust/issues/89626
(That problem may have been fixed in https://github.com/rust-lang/rust/commit/9a04ae4997493e9260352064163285cddc43de3c,
but even in the version containing that patch, [there is report](https://github.com/rust-lang/rust/issues/89626#issuecomment-1242636038)
of the same error.)
+This seems to be due to the fact that compiler-builtins is built before libc (which is a dependency
+of std) links musl. And the std and its dependent can use getauxval without this problem at least
+since the rust-lang/rust patch mentioned above:
+https://github.com/rust-lang/rust/blob/1.85.0/library/std/src/sys/pal/unix/stack_overflow.rs#L268
+(According to https://github.com/rust-lang/rust/issues/89626#issuecomment-2420469392, this problem
+may have been fixed in https://github.com/rust-lang/rust/commit/9ed0d11efbec18a1fa4155576a3bcb685676d23c.)
+See also https://github.com/rust-lang/stdarch/pull/1746.
+So as for musl with static linking, we assume that getauxval is always available also when `std` feature enabled.
On platforms that we cannot assume that getauxval/elf_aux_info is always available, so we use dlsym
instead of directly calling getauxval/elf_aux_info. (You can force getauxval/elf_aux_info to be
@@ -113,28 +147,36 @@ include!("common.rs");
use self::os::ffi;
#[cfg(any(target_os = "linux", target_os = "android"))]
mod os {
- // core::ffi::c_* (except c_void) requires Rust 1.64, libc requires Rust 1.63
+ // libc requires Rust 1.63
#[cfg_attr(test, allow(dead_code))]
pub(super) mod ffi {
- pub(crate) use super::super::c_types::c_ulong;
+ pub(crate) use crate::utils::ffi::c_ulong;
#[allow(unused_imports)]
- pub(crate) use super::super::c_types::{c_char, c_int, c_void};
+ pub(crate) use crate::utils::ffi::{c_char, c_int, c_void};
sys_const!({
- // https://github.com/torvalds/linux/blob/v6.11/include/uapi/linux/auxvec.h
+ // https://github.com/torvalds/linux/blob/v6.16/include/uapi/linux/auxvec.h
pub(crate) const AT_HWCAP: c_ulong = 16;
#[cfg(any(
test,
all(target_arch = "aarch64", target_pointer_width = "64"),
target_arch = "powerpc64",
))]
pub(crate) const AT_HWCAP2: c_ulong = 26;
+ // Linux 6.9+
+ // https://github.com/torvalds/linux/commit/3281366a8e79a512956382885091565db1036b64
+ #[cfg(test)]
+ #[cfg(not(all(target_arch = "aarch64", target_pointer_width = "32")))]
+ pub(crate) const AT_HWCAP3: c_ulong = 29;
+ #[cfg(test)]
+ #[cfg(not(all(target_arch = "aarch64", target_pointer_width = "32")))]
+ pub(crate) const AT_HWCAP4: c_ulong = 30;
// Defined in dlfcn.h.
// https://github.com/bminor/glibc/blob/glibc-2.40/dlfcn/dlfcn.h
// https://github.com/bminor/musl/blob/v1.2.5/include/dlfcn.h
// https://github.com/wbx-github/uclibc-ng/blob/v1.0.47/include/dlfcn.h
- // https://github.com/aosp-mirror/platform_bionic/blob/android-15.0.0_r1/libc/include/dlfcn.h
+ // https://github.com/aosp-mirror/platform_bionic/blob/android-16.0.0_r1/libc/include/dlfcn.h
#[cfg(any(
test,
not(any(
@@ -156,10 +198,15 @@ mod os {
portable_atomic_outline_atomics,
)),
))]
+ #[cfg(not(all(target_os = "android", target_pointer_width = "32")))]
pub(crate) const RTLD_DEFAULT: *mut c_void = core::ptr::null_mut();
+ #[cfg(all(target_os = "android", target_pointer_width = "32"))]
+ #[allow(clippy::cast_sign_loss)]
+ pub(crate) const RTLD_DEFAULT: *mut c_void =
+ crate::utils::ptr::without_provenance_mut(-1_isize as usize);
// Defined in sys/system_properties.h.
- // https://github.com/aosp-mirror/platform_bionic/blob/android-15.0.0_r1/libc/include/sys/system_properties.h
+ // https://github.com/aosp-mirror/platform_bionic/blob/android-16.0.0_r1/libc/include/sys/system_properties.h
#[cfg(all(target_arch = "aarch64", target_os = "android"))]
pub(crate) const PROP_VALUE_MAX: c_int = 92;
});
@@ -171,8 +218,9 @@ mod os {
// https://github.com/bminor/glibc/blob/glibc-2.40/misc/sys/auxv.h
// https://github.com/bminor/musl/blob/v1.2.5/include/sys/auxv.h
// https://github.com/wbx-github/uclibc-ng/blob/v1.0.47/include/sys/auxv.h
- // https://github.com/aosp-mirror/platform_bionic/blob/android-15.0.0_r1/libc/include/sys/auxv.h
- // https://github.com/picolibc/picolibc/blob/1.8.6/newlib/libc/include/sys/auxv.h
+ // https://github.com/kernkonzept/l4re-core/blob/4351d4474804636122d64ea5a5d41f5e78e9208e/uclibc/lib/contrib/uclibc/include/sys/auxv.h
+ // https://github.com/aosp-mirror/platform_bionic/blob/android-16.0.0_r1/libc/include/sys/auxv.h
+ // https://github.com/picolibc/picolibc/blob/1.8.10/newlib/libc/include/sys/auxv.h
#[cfg(any(
test,
all(
@@ -199,7 +247,7 @@ mod os {
// https://github.com/bminor/glibc/blob/glibc-2.40/dlfcn/dlfcn.h
// https://github.com/bminor/musl/blob/v1.2.5/include/dlfcn.h
// https://github.com/wbx-github/uclibc-ng/blob/v1.0.47/include/dlfcn.h
- // https://github.com/aosp-mirror/platform_bionic/blob/android-15.0.0_r1/libc/include/dlfcn.h
+ // https://github.com/aosp-mirror/platform_bionic/blob/android-16.0.0_r1/libc/include/dlfcn.h
#[cfg(any(
test,
not(any(
@@ -224,7 +272,7 @@ mod os {
pub(crate) fn dlsym(handle: *mut c_void, symbol: *const c_char) -> *mut c_void;
// Defined in sys/system_properties.h.
- // https://github.com/aosp-mirror/platform_bionic/blob/android-15.0.0_r1/libc/include/sys/system_properties.h
+ // https://github.com/aosp-mirror/platform_bionic/blob/android-16.0.0_r1/libc/include/sys/system_properties.h
#[cfg(all(target_arch = "aarch64", target_os = "android"))]
pub(crate) fn __system_property_get(
name: *const c_char,
@@ -236,52 +284,41 @@ mod os {
pub(super) type GetauxvalTy = unsafe extern "C" fn(ffi::c_ulong) -> ffi::c_ulong;
pub(super) fn getauxval(type_: ffi::c_ulong) -> ffi::c_ulong {
- #[cfg(any(
- all(
- target_os = "linux",
- any(
- all(
- target_env = "gnu",
- any(
- target_arch = "aarch64",
- all(target_arch = "powerpc64", target_endian = "little"),
- ),
- ),
- target_env = "musl",
- target_env = "ohos",
- ),
- ),
- all(target_os = "android", target_pointer_width = "64"),
- portable_atomic_outline_atomics,
- ))]
- let getauxval: GetauxvalTy = ffi::getauxval;
- #[cfg(not(any(
- all(
- target_os = "linux",
- any(
- all(
- target_env = "gnu",
- any(
- target_arch = "aarch64",
- all(target_arch = "powerpc64", target_endian = "little"),
+ cfg_sel!({
+ #[cfg(any(
+ all(
+ target_os = "linux",
+ any(
+ all(
+ target_env = "gnu",
+ any(
+ target_arch = "aarch64",
+ all(target_arch = "powerpc64", target_endian = "little"),
+ ),
),
+ target_env = "musl",
+ target_env = "ohos",
),
- target_env = "musl",
- target_env = "ohos",
),
- ),
- all(target_os = "android", target_pointer_width = "64"),
- portable_atomic_outline_atomics,
- )))]
- // SAFETY: we passed a valid C string to dlsym, and a pointer returned by dlsym
- // is a valid pointer to the function if it is non-null.
- let getauxval: GetauxvalTy = unsafe {
- let ptr = ffi::dlsym(ffi::RTLD_DEFAULT, "getauxval\0".as_ptr().cast::<ffi::c_char>());
- if ptr.is_null() {
- return 0;
+ all(target_os = "android", target_pointer_width = "64"),
+ portable_atomic_outline_atomics,
+ ))]
+ {
+ let getauxval: GetauxvalTy = ffi::getauxval;
}
- core::mem::transmute::<*mut ffi::c_void, GetauxvalTy>(ptr)
- };
+ #[cfg(else)]
+ {
+ // SAFETY: we passed a valid C string to dlsym, and a pointer returned by dlsym
+ // is a valid pointer to the function if it is non-null.
+ let getauxval: GetauxvalTy = unsafe {
+ let ptr = ffi::dlsym(ffi::RTLD_DEFAULT, c!("getauxval").as_ptr());
+ if ptr.is_null() {
+ return 0;
+ }
+ core::mem::transmute::<*mut ffi::c_void, GetauxvalTy>(ptr)
+ };
+ }
+ });
// SAFETY: `getauxval` is thread-safe.
unsafe { getauxval(type_) }
@@ -291,36 +328,54 @@ mod os {
mod os {
use core::mem;
- // core::ffi::c_* (except c_void) requires Rust 1.64, libc requires Rust 1.63
+ // libc requires Rust 1.63
#[cfg_attr(test, allow(dead_code))]
pub(super) mod ffi {
#[allow(unused_imports)]
- pub(crate) use super::super::c_types::c_char;
- pub(crate) use super::super::c_types::{c_int, c_ulong, c_void};
+ pub(crate) use crate::utils::ffi::c_char;
+ pub(crate) use crate::utils::ffi::{c_int, c_ulong, c_void};
sys_const!({
// FreeBSD
// Defined in sys/elf_common.h.
- // https://github.com/freebsd/freebsd-src/blob/release/14.1.0/sys/sys/elf_common.h
+ // https://github.com/freebsd/freebsd-src/blob/release/14.3.0/sys/sys/elf_common.h
// OpenBSD
// Defined in sys/auxv.h.
// https://github.com/openbsd/src/blob/ed8f5e8d82ace15e4cefca2c82941b15cb1a7830/sys/sys/auxv.h
pub(crate) const AT_HWCAP: c_int = 25;
- #[cfg(any(test, target_arch = "powerpc64"))]
+ #[cfg(any(
+ test,
+ all(target_os = "freebsd", target_arch = "aarch64", target_pointer_width = "64"),
+ target_arch = "powerpc64",
+ ))]
pub(crate) const AT_HWCAP2: c_int = 26;
+ // FreeBSD 15.0+
+ // https://github.com/freebsd/freebsd-src/commit/85007872d1227006adf2ce119fe30de856cbe12d
+ #[cfg(test)]
+ #[cfg(not(target_os = "openbsd"))]
+ pub(crate) const AT_HWCAP3: c_int = 38;
+ #[cfg(test)]
+ #[cfg(not(target_os = "openbsd"))]
+ pub(crate) const AT_HWCAP4: c_int = 39;
// FreeBSD
// Defined in dlfcn.h.
// https://man.freebsd.org/dlsym(3)
- // https://github.com/freebsd/freebsd-src/blob/release/14.1.0/include/dlfcn.h
+ // https://github.com/freebsd/freebsd-src/blob/release/14.3.0/include/dlfcn.h
// OpenBSD
// Defined in dlfcn.h.
// https://man.openbsd.org/dlsym.3
// https://github.com/openbsd/src/blob/ed8f5e8d82ace15e4cefca2c82941b15cb1a7830/include/dlfcn.h
#[cfg(any(
test,
not(any(
- all(target_os = "freebsd", target_arch = "aarch64"),
+ all(
+ target_os = "freebsd",
+ any(
+ target_arch = "aarch64",
+ all(target_arch = "powerpc64", target_endian = "little"),
+ ),
+ ),
portable_atomic_outline_atomics,
)),
))]
@@ -333,15 +388,21 @@ mod os {
// FreeBSD
// Defined in sys/auxv.h.
// https://man.freebsd.org/elf_aux_info(3)
- // https://github.com/freebsd/freebsd-src/blob/release/14.1.0/sys/sys/auxv.h
+ // https://github.com/freebsd/freebsd-src/blob/release/14.3.0/sys/sys/auxv.h
// OpenBSD
// Defined in sys/auxv.h.
// https://man.openbsd.org/elf_aux_info.3
// https://github.com/openbsd/src/blob/ed8f5e8d82ace15e4cefca2c82941b15cb1a7830/sys/sys/auxv.h
#[cfg(any(
test,
any(
- all(target_os = "freebsd", target_arch = "aarch64"),
+ all(
+ target_os = "freebsd",
+ any(
+ target_arch = "aarch64",
+ all(target_arch = "powerpc64", target_endian = "little"),
+ ),
+ ),
portable_atomic_outline_atomics,
),
))]
@@ -350,15 +411,21 @@ mod os {
// FreeBSD
// Defined in dlfcn.h.
// https://man.freebsd.org/dlsym(3)
- // https://github.com/freebsd/freebsd-src/blob/release/14.1.0/include/dlfcn.h
+ // https://github.com/freebsd/freebsd-src/blob/release/14.3.0/include/dlfcn.h
// OpenBSD
// Defined in dlfcn.h.
// https://man.openbsd.org/dlsym.3
// https://github.com/openbsd/src/blob/ed8f5e8d82ace15e4cefca2c82941b15cb1a7830/include/dlfcn.h
#[cfg(any(
test,
not(any(
- all(target_os = "freebsd", target_arch = "aarch64"),
+ all(
+ target_os = "freebsd",
+ any(
+ target_arch = "aarch64",
+ all(target_arch = "powerpc64", target_endian = "little"),
+ ),
+ ),
portable_atomic_outline_atomics,
)),
))]
@@ -373,25 +440,33 @@ mod os {
#[allow(clippy::cast_possible_wrap, clippy::cast_possible_truncation)]
const OUT_LEN: ffi::c_int = mem::size_of::<ffi::c_ulong>() as ffi::c_int;
- #[cfg(any(
- all(target_os = "freebsd", target_arch = "aarch64"),
- portable_atomic_outline_atomics,
- ))]
- let elf_aux_info: ElfAuxInfoTy = ffi::elf_aux_info;
- #[cfg(not(any(
- all(target_os = "freebsd", target_arch = "aarch64"),
- portable_atomic_outline_atomics,
- )))]
- // SAFETY: we passed a valid C string to dlsym, and a pointer returned by dlsym
- // is a valid pointer to the function if it is non-null.
- let elf_aux_info: ElfAuxInfoTy = unsafe {
- let ptr =
- ffi::dlsym(ffi::RTLD_DEFAULT, "elf_aux_info\0".as_ptr().cast::<ffi::c_char>());
- if ptr.is_null() {
- return 0;
+ cfg_sel!({
+ #[cfg(any(
+ all(
+ target_os = "freebsd",
+ any(
+ target_arch = "aarch64",
+ all(target_arch = "powerpc64", target_endian = "little"),
+ ),
+ ),
+ portable_atomic_outline_atomics,
+ ))]
+ {
+ let elf_aux_info: ElfAuxInfoTy = ffi::elf_aux_info;
}
- mem::transmute::<*mut ffi::c_void, ElfAuxInfoTy>(ptr)
- };
+ #[cfg(else)]
+ {
+ // SAFETY: we passed a valid C string to dlsym, and a pointer returned by dlsym
+ // is a valid pointer to the function if it is non-null.
+ let elf_aux_info: ElfAuxInfoTy = unsafe {
+ let ptr = ffi::dlsym(ffi::RTLD_DEFAULT, c!("elf_aux_info").as_ptr());
+ if ptr.is_null() {
+ return 0;
+ }
+ mem::transmute::<*mut ffi::c_void, ElfAuxInfoTy>(ptr)
+ };
+ }
+ });
let mut out: ffi::c_ulong = 0;
// SAFETY:
@@ -412,15 +487,15 @@ mod os {
use self::arch::_detect;
#[cfg(target_arch = "aarch64")]
mod arch {
- use super::{ffi, os, CpuInfo};
+ use super::{CpuInfo, CpuInfoFlag, ffi, os};
sys_const!({
// Linux
- // https://github.com/torvalds/linux/blob/v6.11/arch/arm64/include/uapi/asm/hwcap.h
- // https://github.com/torvalds/linux/blob/v6.11/Documentation/arch/arm64/elf_hwcaps.rst
+ // https://github.com/torvalds/linux/blob/v6.16/arch/arm64/include/uapi/asm/hwcap.h
+ // https://github.com/torvalds/linux/blob/v6.16/Documentation/arch/arm64/elf_hwcaps.rst
// FreeBSD
// Defined in machine/elf.h.
- // https://github.com/freebsd/freebsd-src/blob/release/14.1.0/sys/arm64/include/elf.h
+ // https://github.com/freebsd/freebsd-src/blob/release/14.3.0/sys/arm64/include/elf.h
// OpenBSD
// Defined in machine/elf.h.
// https://github.com/openbsd/src/blob/ed8f5e8d82ace15e4cefca2c82941b15cb1a7830/sys/arch/arm64/include/elf.h
@@ -431,25 +506,55 @@ mod arch {
// https://github.com/freebsd/freebsd-src/blob/release/12.2.0/sys/arm64/include/elf.h
// OpenBSD 7.6+
// https://github.com/openbsd/src/commit/ef873df06dac50249b2dd380dc6100eee3b0d23d
- pub(super) const HWCAP_ATOMICS: ffi::c_ulong = 1 << 8;
+ pub(crate) const HWCAP_ATOMICS: ffi::c_ulong = 1 << 8;
+ // Linux 4.11+
+ // https://github.com/torvalds/linux/commit/77c97b4ee21290f5f083173d957843b615abbff2
+ // FreeBSD 13.0+/12.2+
+ // https://github.com/freebsd/freebsd-src/blob/release/13.0.0/sys/arm64/include/elf.h
+ // https://github.com/freebsd/freebsd-src/blob/release/12.2.0/sys/arm64/include/elf.h
+ // OpenBSD 7.6+
+ // https://github.com/openbsd/src/commit/ef873df06dac50249b2dd380dc6100eee3b0d23d
+ #[cfg(test)]
+ pub(crate) const HWCAP_CPUID: ffi::c_ulong = 1 << 11;
+ // Linux 4.12+
+ // https://github.com/torvalds/linux/commit/c651aae5a7732287c1c9bc974ece4ed798780544
+ // FreeBSD 13.0+/12.2+
+ // https://github.com/freebsd/freebsd-src/blob/release/13.0.0/sys/arm64/include/elf.h
+ // https://github.com/freebsd/freebsd-src/blob/release/12.2.0/sys/arm64/include/elf.h
+ // OpenBSD 7.6+
+ // https://github.com/openbsd/src/commit/ef873df06dac50249b2dd380dc6100eee3b0d23d
+ #[cfg(test)]
+ pub(crate) const HWCAP_LRCPC: ffi::c_ulong = 1 << 15;
// Linux 4.17+
// https://github.com/torvalds/linux/commit/7206dc93a58fb76421c4411eefa3c003337bcb2d
// FreeBSD 13.0+/12.2+
// https://github.com/freebsd/freebsd-src/blob/release/13.0.0/sys/arm64/include/elf.h
// https://github.com/freebsd/freebsd-src/blob/release/12.2.0/sys/arm64/include/elf.h
// OpenBSD 7.6+
// https://github.com/openbsd/src/commit/ef873df06dac50249b2dd380dc6100eee3b0d23d
- pub(super) const HWCAP_USCAT: ffi::c_ulong = 1 << 25;
+ pub(crate) const HWCAP_USCAT: ffi::c_ulong = 1 << 25;
+ #[cfg(test)]
+ pub(crate) const HWCAP_ILRCPC: ffi::c_ulong = 1 << 26;
// Linux 6.7+
// https://github.com/torvalds/linux/commit/338a835f40a849cd89b993e342bd9fbd5684825c
- #[cfg(any(target_os = "linux", target_os = "android"))]
+ // FreeBSD 15.0+
+ // https://github.com/freebsd/freebsd-src/commit/94686b081fdb0c1bb0fc1dfeda14bd53f26ce7c5
+ #[cfg(not(target_os = "openbsd"))]
#[cfg(target_pointer_width = "64")]
- pub(super) const HWCAP2_LRCPC3: ffi::c_ulong = 1 << 46;
+ pub(crate) const HWCAP2_LRCPC3: ffi::c_ulong = 1 << 46;
// Linux 6.7+
// https://github.com/torvalds/linux/commit/94d0657f9f0d311489606589133ebf49e28104d8
+ // FreeBSD 15.0+
+ // https://github.com/freebsd/freebsd-src/commit/94686b081fdb0c1bb0fc1dfeda14bd53f26ce7c5
+ #[cfg(not(target_os = "openbsd"))]
+ #[cfg(target_pointer_width = "64")]
+ pub(crate) const HWCAP2_LSE128: ffi::c_ulong = 1 << 47;
+ // Linux 6.18+
+ // https://github.com/torvalds/linux/commit/220928e52cb03d223b3acad3888baf0687486d21
+ #[cfg(test)]
#[cfg(any(target_os = "linux", target_os = "android"))]
#[cfg(target_pointer_width = "64")]
- pub(super) const HWCAP2_LSE128: ffi::c_ulong = 1 << 47;
+ pub(crate) const HWCAP3_LSFE: ffi::c_ulong = 1 << 2;
});
#[cold]
@@ -464,7 +569,7 @@ mod arch {
// SAFETY: we've passed a valid C string and a buffer with max length.
let len = unsafe {
ffi::__system_property_get(
- b"ro.arch\0".as_ptr().cast::<ffi::c_char>(),
+ c!("ro.arch").as_ptr(),
arch.as_mut_ptr().cast::<ffi::c_char>(),
)
};
@@ -475,39 +580,92 @@ mod arch {
}
}
- let hwcap = os::getauxval(ffi::AT_HWCAP);
-
- if hwcap & HWCAP_ATOMICS != 0 {
- info.set(CpuInfo::HAS_LSE);
- }
- if hwcap & HWCAP_USCAT != 0 {
- info.set(CpuInfo::HAS_LSE2);
+ macro_rules! check {
+ ($x:ident, $flag:ident, $bit:ident) => {
+ if $x & $bit != 0 {
+ info.set(CpuInfoFlag::$flag);
+ }
+ };
}
- #[cfg(any(target_os = "linux", target_os = "android"))]
+ let hwcap = os::getauxval(ffi::AT_HWCAP);
+ check!(hwcap, lse, HWCAP_ATOMICS);
+ check!(hwcap, lse2, HWCAP_USCAT);
+ #[cfg(test)]
+ check!(hwcap, rcpc, HWCAP_LRCPC);
+ #[cfg(test)]
+ check!(hwcap, rcpc2, HWCAP_ILRCPC);
+ #[cfg(test)]
+ check!(hwcap, cpuid, HWCAP_CPUID);
+ #[cfg(not(target_os = "openbsd"))]
// HWCAP2 is not yet available on ILP32: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/tree/arch/arm64/include/uapi/asm/hwcap.h?h=staging/ilp32-5.1
#[cfg(target_pointer_width = "64")]
{
let hwcap2 = os::getauxval(ffi::AT_HWCAP2);
- if hwcap2 & HWCAP2_LRCPC3 != 0 {
- info.set(CpuInfo::HAS_RCPC3);
- }
- if hwcap2 & HWCAP2_LSE128 != 0 {
- info.set(CpuInfo::HAS_LSE128);
+ check!(hwcap2, rcpc3, HWCAP2_LRCPC3);
+ check!(hwcap2, lse128, HWCAP2_LSE128);
+ #[cfg(test)]
+ #[cfg(any(target_os = "linux", target_os = "android"))]
+ {
+ let hwcap3 = os::getauxval(ffi::AT_HWCAP3);
+ check!(hwcap3, lsfe, HWCAP3_LSFE);
}
}
}
}
+#[cfg(target_arch = "arm")]
+mod arch {
+ use super::{CpuInfo, CpuInfoFlag, ffi, os};
+
+ sys_const!({
+ // Linux
+ // https://github.com/torvalds/linux/blob/v6.16/arch/arm/include/uapi/asm/hwcap.h
+ // FreeBSD
+ // Defined in machine/elf.h.
+ // https://github.com/freebsd/freebsd-src/blob/release/14.3.0/sys/arm/include/elf.h
+ // OpenBSD
+ // Defined in machine/elf.h.
+ // https://github.com/openbsd/src/blob/ed8f5e8d82ace15e4cefca2c82941b15cb1a7830/sys/arch/arm/include/elf.h
+ // Linux 3.11+
+ // https://github.com/torvalds/linux/commit/a469abd0f868c902b75532579bf87553dcf1b360
+ // FreeBSD 12.0+/11.2+
+ // https://github.com/freebsd/freebsd-src/commit/0cbf724ed03571bc90ed22c3b4bf8c6c7b2da564
+ // https://github.com/freebsd/freebsd-src/blob/release/11.2.0/sys/arm/include/elf.h
+ // OpenBSD 7.6+
+ // https://github.com/openbsd/src/commit/ef873df06dac50249b2dd380dc6100eee3b0d23d
+ #[cfg(test)]
+ pub(crate) const HWCAP_LPAE: ffi::c_ulong = 1 << 20;
+ });
+
+ #[cold]
+ pub(crate) fn _detect(info: &mut CpuInfo) {
+ macro_rules! check {
+ ($x:ident, $flag:ident, $($bit:ident) ||+) => {
+ if $x & ($($bit) |+) != 0 {
+ info.set(CpuInfoFlag::$flag);
+ }
+ };
+ ($x:ident, $flag:ident, $($bit:ident) &&+) => {
+ if $x & ($($bit) |+) == ($($bit) |+) {
+ info.set(CpuInfoFlag::$flag);
+ }
+ };
+ }
+ let hwcap = os::getauxval(ffi::AT_HWCAP);
+ #[cfg(test)]
+ check!(hwcap, lpae, HWCAP_LPAE);
+ }
+}
#[cfg(target_arch = "powerpc64")]
mod arch {
- use super::{ffi, os, CpuInfo};
+ use super::{CpuInfo, CpuInfoFlag, ffi, os};
sys_const!({
// Linux
- // https://github.com/torvalds/linux/blob/v6.11/arch/powerpc/include/uapi/asm/cputable.h
- // https://github.com/torvalds/linux/blob/v6.11/Documentation/arch/powerpc/elf_hwcaps.rst
+ // https://github.com/torvalds/linux/blob/v6.16/arch/powerpc/include/uapi/asm/cputable.h
+ // https://github.com/torvalds/linux/blob/v6.16/Documentation/arch/powerpc/elf_hwcaps.rst
// FreeBSD
// Defined in machine/cpu.h.
- // https://github.com/freebsd/freebsd-src/blob/release/14.1.0/sys/powerpc/include/cpu.h
+ // https://github.com/freebsd/freebsd-src/blob/release/14.3.0/sys/powerpc/include/cpu.h
// OpenBSD
// Defined in machine/elf.h.
// https://github.com/openbsd/src/blob/ed8f5e8d82ace15e4cefca2c82941b15cb1a7830/sys/arch/powerpc64/include/elf.h
@@ -517,27 +675,29 @@ mod arch {
// https://github.com/freebsd/freebsd-src/commit/b0bf7fcd298133457991b27625bbed766e612730
// OpenBSD 7.6+
// https://github.com/openbsd/src/commit/0b0568a19fc4c197871ceafbabc91fabf17ca152
- pub(super) const PPC_FEATURE_BOOKE: ffi::c_ulong = 0x00008000;
+ pub(crate) const PPC_FEATURE_BOOKE: ffi::c_ulong = 0x00008000;
// Linux 3.10+
// https://github.com/torvalds/linux/commit/cbbc6f1b1433ef553d57826eee87a84ca49645ce
// FreeBSD 11.0+
// https://github.com/freebsd/freebsd-src/commit/b0bf7fcd298133457991b27625bbed766e612730
// OpenBSD 7.6+
// https://github.com/openbsd/src/commit/0b0568a19fc4c197871ceafbabc91fabf17ca152
- pub(super) const PPC_FEATURE2_ARCH_2_07: ffi::c_ulong = 0x80000000;
+ pub(crate) const PPC_FEATURE2_ARCH_2_07: ffi::c_ulong = 0x80000000;
// Linux 4.5+
// https://github.com/torvalds/linux/commit/e708c24cd01ce80b1609d8baccee40ccc3608a01
// FreeBSD 12.0+
// https://github.com/freebsd/freebsd-src/commit/18f48e0c72f91bc2d4373078a3f1ab1bcab4d8b3
// OpenBSD 7.6+
// https://github.com/openbsd/src/commit/0b0568a19fc4c197871ceafbabc91fabf17ca152
- pub(super) const PPC_FEATURE2_ARCH_3_00: ffi::c_ulong = 0x00800000;
+ pub(crate) const PPC_FEATURE2_ARCH_3_00: ffi::c_ulong = 0x00800000;
// Linux 5.8+
// https://github.com/torvalds/linux/commit/ee988c11acf6f9464b7b44e9a091bf6afb3b3a49
- // FreeBSD 15.0+
+ // FreeBSD 15.0+/14.2+
// https://github.com/freebsd/freebsd-src/commit/1e434da3b065ef96b389e5e0b604ae05a51e794e
- #[cfg(not(target_os = "openbsd"))]
- pub(super) const PPC_FEATURE2_ARCH_3_1: ffi::c_ulong = 0x00040000;
+ // https://github.com/freebsd/freebsd-src/blob/release/14.2.0/sys/powerpc/include/cpu.h
+ // OpenBSD 7.7+
+ // https://github.com/openbsd/src/commit/483a78e15aaa23c010911940770c1c97db5c1287
+ pub(crate) const PPC_FEATURE2_ARCH_3_1: ffi::c_ulong = 0x00040000;
});
#[cold]
@@ -555,15 +715,12 @@ mod arch {
let hwcap2 = os::getauxval(ffi::AT_HWCAP2);
// Check both 2_07 and later ISAs (which are superset of 2_07) because
// OpenBSD currently doesn't set 2_07 even when 3_00 is set.
- // https://github.com/openbsd/src/blob/ed8f5e8d82ace15e4cefca2c82941b15cb1a7830/sys/arch/powerpc64/powerpc64/cpu.c#L224-L243
+ // https://github.com/openbsd/src/blob/d8ec5edcdf1fb224619831ad90668c95e45c3e36/sys/arch/powerpc64/powerpc64/cpu.c#L222-L238
// Other OSes should be fine, but check all OSs in the same way just in case.
- #[cfg(not(target_os = "openbsd"))]
let isa_2_07_or_later =
PPC_FEATURE2_ARCH_2_07 | PPC_FEATURE2_ARCH_3_00 | PPC_FEATURE2_ARCH_3_1;
- #[cfg(target_os = "openbsd")]
- let isa_2_07_or_later = PPC_FEATURE2_ARCH_2_07 | PPC_FEATURE2_ARCH_3_00;
if hwcap2 & isa_2_07_or_later != 0 {
- info.set(CpuInfo::HAS_QUADWORD_ATOMICS);
+ info.set(CpuInfoFlag::quadword_atomics);
}
}
}
@@ -585,19 +742,18 @@ mod tests {
#[cfg(all(target_arch = "aarch64", target_os = "android"))]
#[test]
fn test_android() {
+ use std::{slice, str};
unsafe {
let mut arch = [1; ffi::PROP_VALUE_MAX as usize];
let len = ffi::__system_property_get(
- b"ro.arch\0".as_ptr().cast::<ffi::c_char>(),
+ c!("ro.arch").as_ptr(),
arch.as_mut_ptr().cast::<ffi::c_char>(),
);
assert!(len >= 0);
- std::eprintln!("len={}", len);
- std::eprintln!("arch={:?}", arch);
- std::eprintln!(
- "arch={:?}",
- core::str::from_utf8(core::slice::from_raw_parts(arch.as_ptr(), len as usize))
- .unwrap()
+ test_helper::eprintln_nocapture!("ro.arch=raw={:?},len={}", arch, len);
+ test_helper::eprintln_nocapture!(
+ "ro.arch={:?}",
+ str::from_utf8(slice::from_raw_parts(arch.as_ptr(), len as usize)).unwrap()
);
}
}
@@ -606,17 +762,13 @@ mod tests {
#[test]
fn test_dlsym_getauxval() {
unsafe {
- let ptr = ffi::dlsym(ffi::RTLD_DEFAULT, "getauxval\0".as_ptr().cast::<ffi::c_char>());
- if cfg!(any(
+ let ptr = ffi::dlsym(ffi::RTLD_DEFAULT, c!("getauxval").as_ptr());
+ if cfg!(target_feature = "crt-static") {
+ assert!(ptr.is_null());
+ } else if cfg!(any(
all(
target_os = "linux",
- any(
- target_env = "gnu",
- all(
- any(target_env = "musl", target_env = "ohos"),
- not(target_feature = "crt-static"),
- ),
- ),
+ any(target_env = "gnu", target_env = "musl", target_env = "ohos"),
),
target_os = "android",
)) {
@@ -628,114 +780,160 @@ mod tests {
return;
}
let dlsym_getauxval = mem::transmute::<*mut ffi::c_void, os::GetauxvalTy>(ptr);
- assert_eq!(dlsym_getauxval(ffi::AT_HWCAP), ffi::getauxval(ffi::AT_HWCAP));
- assert_eq!(dlsym_getauxval(ffi::AT_HWCAP2), ffi::getauxval(ffi::AT_HWCAP2));
+ for &at in &[ffi::AT_HWCAP, ffi::AT_HWCAP2] {
+ assert_eq!(dlsym_getauxval(at), ffi::getauxval(at));
+ }
+ #[cfg(not(all(target_arch = "aarch64", target_pointer_width = "32")))]
+ for &at in &[ffi::AT_HWCAP3, ffi::AT_HWCAP4] {
+ assert_eq!(dlsym_getauxval(at), ffi::getauxval(at));
+ }
}
}
#[cfg(any(target_os = "freebsd", target_os = "openbsd"))]
#[test]
fn test_dlsym_elf_aux_info() {
unsafe {
- let ptr =
- ffi::dlsym(ffi::RTLD_DEFAULT, "elf_aux_info\0".as_ptr().cast::<ffi::c_char>());
- if cfg!(target_os = "freebsd") || option_env!("CI").is_some() {
+ let ptr = ffi::dlsym(ffi::RTLD_DEFAULT, c!("elf_aux_info").as_ptr());
+ if cfg!(target_feature = "crt-static") {
+ assert!(ptr.is_null());
+ } else if cfg!(target_os = "freebsd") || option_env!("CI").is_some() {
assert!(!ptr.is_null());
}
if ptr.is_null() {
return;
}
let dlsym_elf_aux_info = mem::transmute::<*mut ffi::c_void, os::ElfAuxInfoTy>(ptr);
- let mut out: ffi::c_ulong = 0;
- let mut dlsym_out: ffi::c_ulong = 0;
#[allow(clippy::cast_possible_wrap, clippy::cast_possible_truncation)]
let out_len = mem::size_of::<ffi::c_ulong>() as ffi::c_int;
- assert_eq!(
- ffi::elf_aux_info(
- ffi::AT_HWCAP,
- (&mut out as *mut ffi::c_ulong).cast::<ffi::c_void>(),
- out_len,
- ),
- dlsym_elf_aux_info(
- ffi::AT_HWCAP,
- (&mut dlsym_out as *mut ffi::c_ulong).cast::<ffi::c_void>(),
- out_len,
- ),
- );
- assert_eq!(out, dlsym_out);
- out = 0;
- dlsym_out = 0;
- assert_eq!(
- ffi::elf_aux_info(
- ffi::AT_HWCAP2,
- (&mut out as *mut ffi::c_ulong).cast::<ffi::c_void>(),
- out_len,
- ),
- dlsym_elf_aux_info(
- ffi::AT_HWCAP2,
- (&mut dlsym_out as *mut ffi::c_ulong).cast::<ffi::c_void>(),
- out_len,
- ),
- );
- assert_eq!(out, dlsym_out);
+ for &at in &[ffi::AT_HWCAP, ffi::AT_HWCAP2] {
+ let mut out: ffi::c_ulong = 0;
+ let mut dlsym_out: ffi::c_ulong = 0;
+ assert_eq!(
+ ffi::elf_aux_info(
+ at,
+ (&mut out as *mut ffi::c_ulong).cast::<ffi::c_void>(),
+ out_len,
+ ),
+ dlsym_elf_aux_info(
+ at,
+ (&mut dlsym_out as *mut ffi::c_ulong).cast::<ffi::c_void>(),
+ out_len,
+ ),
+ );
+ assert_eq!(out, dlsym_out);
+ }
+ #[cfg(not(target_os = "openbsd"))]
+ for &at in &[ffi::AT_HWCAP3, ffi::AT_HWCAP4] {
+ let mut out: ffi::c_ulong = 0;
+ let mut dlsym_out: ffi::c_ulong = 0;
+ assert_eq!(
+ ffi::elf_aux_info(
+ at,
+ (&mut out as *mut ffi::c_ulong).cast::<ffi::c_void>(),
+ out_len,
+ ),
+ dlsym_elf_aux_info(
+ at,
+ (&mut dlsym_out as *mut ffi::c_ulong).cast::<ffi::c_void>(),
+ out_len,
+ ),
+ );
+ assert_eq!(out, dlsym_out);
+ }
}
}
#[cfg(any(target_os = "linux", target_os = "android"))]
#[cfg(not(all(target_arch = "aarch64", target_pointer_width = "32")))]
+ #[cfg_attr(target_arch = "arm", rustversion::nightly)] // cfg(target_feature = "thumb-mode") is nightly-only
+ #[cfg_attr(target_arch = "powerpc64", rustversion::since(1.92))] // requires https://github.com/rust-lang/rust/pull/146831
#[test]
fn test_alternative() {
- use c_types::*;
#[cfg(not(portable_atomic_no_asm))]
use std::arch::asm;
use std::{str, vec};
+
#[cfg(target_pointer_width = "32")]
use sys::Elf32_auxv_t as Elf_auxv_t;
#[cfg(target_pointer_width = "64")]
use sys::Elf64_auxv_t as Elf_auxv_t;
use test_helper::sys;
+ use crate::utils::{RegISize, RegSize, ffi::*};
+
// Linux kernel 6.4 has added a way to read auxv without depending on either libc or mrs trap.
// https://github.com/torvalds/linux/commit/ddc65971bb677aa9f6a4c21f76d3133e106f88eb
// (Actually 6.5? https://github.com/torvalds/linux/commit/636e348353a7cc52609fdba5ff3270065da140d5)
//
// This is currently used only for testing.
fn getauxval_pr_get_auxv_no_libc(type_: c_ulong) -> Result<c_ulong, c_int> {
- #[cfg(target_arch = "aarch64")]
+ // Refs:
+ // - aarch64
+ // https://github.com/bminor/musl/blob/v1.2.5/arch/aarch64/syscall_arch.h
+ // - arm
+ // https://github.com/bminor/musl/blob/v1.2.5/arch/arm/syscall_arch.h
+ // - powerpc64
+ // https://github.com/torvalds/linux/blob/v6.18/Documentation/arch/powerpc/syscall64-abi.rst
+ // https://github.com/bminor/musl/blob/1b76ff0767d01df72f692806ee5adee13c67ef88/arch/powerpc64/syscall_arch.h
unsafe fn prctl_get_auxv(out: *mut c_void, len: usize) -> Result<usize, c_int> {
- let r: i64;
+ // arguments must be extended to 64-bit if 64-bit arch
+ let number = sys::__NR_prctl as RegSize;
+ let arg1 = sys::PR_GET_AUXV as RegSize;
+ let arg2 = ptr_reg!(out);
+ let arg3 = len as RegSize;
+ let r: RegISize;
unsafe {
+ #[cfg(target_arch = "aarch64")]
asm!(
"svc 0",
- in("x8") sys::__NR_prctl as u64,
- inout("x0") sys::PR_GET_AUXV as u64 => r,
- in("x1") ptr_reg!(out),
- in("x2") len as u64,
+ in("x8") number,
+ inout("x0") arg1 => r,
+ in("x1") arg2,
+ in("x2") arg3,
// arg4 and arg5 must be zero.
in("x3") 0_u64,
in("x4") 0_u64,
options(nostack, preserves_flags),
);
- }
- #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)]
- if (r as c_int) < 0 {
- Err(r as c_int)
- } else {
- Ok(r as usize)
- }
- }
- #[cfg(target_arch = "powerpc64")]
- unsafe fn prctl_get_auxv(out: *mut c_void, len: usize) -> Result<usize, c_int> {
- let r: i64;
- unsafe {
+ #[cfg(all(target_arch = "arm", not(target_feature = "thumb-mode")))]
+ asm!(
+ "svc 0",
+ in("r7") number,
+ inout("r0") arg1 => r,
+ in("r1") arg2,
+ in("r2") arg3,
+ // arg4 and arg5 must be zero.
+ in("r3") 0_u32,
+ in("r4") 0_u32,
+ options(nostack, preserves_flags),
+ );
+ #[cfg(all(target_arch = "arm", target_feature = "thumb-mode"))]
+ asm!(
+ // r7 is reserved on thumb
+ "mov {tmp}, r7",
+ "mov r7, {number}",
+ "svc 0",
+ "mov r7, {tmp}",
+ number = in(reg) number,
+ tmp = out(reg) _,
+ inout("r0") arg1 => r,
+ in("r1") arg2,
+ in("r2") arg3,
+ // arg4 and arg5 must be zero.
+ in("r3") 0_u32,
+ in("r4") 0_u32,
+ options(nostack, preserves_flags),
+ );
+ #[cfg(target_arch = "powerpc64")]
asm!(
"sc",
"bns+ 2f",
"neg %r3, %r3",
"2:",
- inout("r0") sys::__NR_prctl as u64 => _,
- inout("r3") sys::PR_GET_AUXV as u64 => r,
- inout("r4") ptr_reg!(out) => _,
- inout("r5") len as u64 => _,
+ inout("r0") number => _,
+ inout("r3") arg1 => r,
+ inout("r4") arg2 => _,
+ inout("r5") arg3 => _,
// arg4 and arg5 must be zero.
inout("r6") 0_u64 => _,
inout("r7") 0_u64 => _,
@@ -745,15 +943,13 @@ mod tests {
out("r11") _,
out("r12") _,
out("cr0") _,
+ out("ctr") _,
+ out("xer") _,
options(nostack, preserves_flags),
);
}
#[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)]
- if (r as c_int) < 0 {
- Err(r as c_int)
- } else {
- Ok(r as usize)
- }
+ if (r as c_int) < 0 { Err(r as c_int) } else { Ok(r as usize) }
}
let mut auxv = vec![unsafe { mem::zeroed::<Elf_auxv_t>() }; 38];
@@ -782,11 +978,7 @@ mod tests {
#[allow(clippy::cast_possible_wrap)]
let r = unsafe { libc::prctl(sys::PR_GET_AUXV as c_int, out, len, 0, 0) };
#[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)]
- if (r as c_int) < 0 {
- Err(r as c_int)
- } else {
- Ok(r as usize)
- }
+ if (r as c_int) < 0 { Err(r as c_int) } else { Ok(r as usize) }
}
let mut auxv = vec![unsafe { mem::zeroed::<Elf_auxv_t>() }; 38];
@@ -815,50 +1007,56 @@ mod tests {
let mut digits = release.split('.');
let major = digits.next().unwrap().parse::<u32>().unwrap();
let minor = digits.next().unwrap().parse::<u32>().unwrap();
- // TODO: qemu-user bug?
+ // TODO: qemu-user bug (fails even on kernel >= 6.4) (as of 9.2)
if (major, minor) < (6, 4) || cfg!(qemu) {
std::eprintln!("kernel version: {}.{} (no pr_get_auxv)", major, minor);
- assert_eq!(getauxval_pr_get_auxv_libc(ffi::AT_HWCAP).unwrap_err(), -1);
- assert_eq!(getauxval_pr_get_auxv_libc(ffi::AT_HWCAP2).unwrap_err(), -1);
- assert_eq!(
- getauxval_pr_get_auxv_no_libc(ffi::AT_HWCAP).unwrap_err(),
- -libc::EINVAL
- );
- assert_eq!(
- getauxval_pr_get_auxv_no_libc(ffi::AT_HWCAP2).unwrap_err(),
- -libc::EINVAL
- );
+ for &at in &[ffi::AT_HWCAP, ffi::AT_HWCAP2, ffi::AT_HWCAP3, ffi::AT_HWCAP4] {
+ assert_eq!(getauxval_pr_get_auxv_libc(at).unwrap_err(), -1);
+ assert_eq!(getauxval_pr_get_auxv_no_libc(at).unwrap_err(), -libc::EINVAL);
+ }
} else {
std::eprintln!("kernel version: {}.{} (has pr_get_auxv)", major, minor);
- assert_eq!(
- os::getauxval(ffi::AT_HWCAP),
- getauxval_pr_get_auxv_libc(ffi::AT_HWCAP).unwrap()
- );
- assert_eq!(
- os::getauxval(ffi::AT_HWCAP2),
- getauxval_pr_get_auxv_libc(ffi::AT_HWCAP2).unwrap()
- );
- assert_eq!(
- os::getauxval(ffi::AT_HWCAP),
- getauxval_pr_get_auxv_no_libc(ffi::AT_HWCAP).unwrap()
- );
- assert_eq!(
- os::getauxval(ffi::AT_HWCAP2),
- getauxval_pr_get_auxv_no_libc(ffi::AT_HWCAP2).unwrap()
- );
+ for &at in &[ffi::AT_HWCAP, ffi::AT_HWCAP2] {
+ if cfg!(all(valgrind, target_arch = "powerpc64")) {
+ // TODO: valgrind bug (as of Valgrind 3.26)
+ assert_eq!(getauxval_pr_get_auxv_libc(at).unwrap_err(), -1);
+ assert_eq!(getauxval_pr_get_auxv_no_libc(at).unwrap_err(), -libc::EINVAL);
+ } else if cfg!(all(valgrind, target_arch = "aarch64"))
+ || cfg!(all(valgrind, target_arch = "arm")) && at == ffi::AT_HWCAP2
+ {
+ // TODO: valgrind bug (result value mismatch) (as of Valgrind 3.26)
+ assert_ne!(os::getauxval(at), getauxval_pr_get_auxv_libc(at).unwrap());
+ assert_ne!(os::getauxval(at), getauxval_pr_get_auxv_no_libc(at).unwrap());
+ } else {
+ assert_eq!(os::getauxval(at), getauxval_pr_get_auxv_libc(at).unwrap());
+ assert_eq!(os::getauxval(at), getauxval_pr_get_auxv_no_libc(at).unwrap());
+ }
+ }
+ for &at in &[ffi::AT_HWCAP3, ffi::AT_HWCAP4] {
+ assert_eq!(
+ os::getauxval(at),
+ getauxval_pr_get_auxv_libc(at).unwrap_or_default()
+ );
+ assert_eq!(
+ os::getauxval(at),
+ getauxval_pr_get_auxv_no_libc(at).unwrap_or_default()
+ );
+ }
}
}
}
#[allow(clippy::cast_possible_wrap)]
#[cfg(target_os = "freebsd")]
#[test]
fn test_alternative() {
- use c_types::*;
#[cfg(not(portable_atomic_no_asm))]
use std::arch::asm;
use std::ptr;
+
use test_helper::sys;
+ use crate::utils::{RegISize, RegSize, ffi::*};
+
// This is almost equivalent to what elf_aux_info does.
// https://man.freebsd.org/elf_aux_info(3)
// On FreeBSD, [AArch64 support is available on FreeBSD 11.0+](https://www.freebsd.org/releases/11.0R/announce),
@@ -906,6 +1104,7 @@ mod tests {
}
for aux in &auxv {
+ #[allow(clippy::cast_sign_loss)]
if aux.a_type == type_ as c_long {
// SAFETY: aux.a_un is #[repr(C)] union and all fields have
// the same size and can be safely transmuted to integers.
@@ -925,75 +1124,30 @@ mod tests {
#[allow(non_camel_case_types)]
type pid_t = c_int;
- // https://github.com/freebsd/freebsd-src/blob/release/14.1.0/lib/libc/aarch64/SYS.h
- // https://github.com/golang/go/blob/4badad8d477ffd7a6b762c35bc69aed82faface7/src/syscall/asm_freebsd_arm64.s
- #[cfg(target_arch = "aarch64")]
+ // Refs:
+ // - aarch64
+ // https://github.com/freebsd/freebsd-src/blob/release/15.0.0/lib/libsys/aarch64/SYS.h
+ // https://github.com/golang/go/blob/go1.25.0/src/syscall/asm_freebsd_arm64.s
+ // - powerpc64
+ // https://github.com/freebsd/freebsd-src/blob/release/15.0.0/lib/libsys/powerpc64/SYS.h
#[inline]
fn getpid() -> pid_t {
- #[allow(clippy::cast_possible_truncation)]
+ let n = sys::SYS_getpid as RegSize;
+ let r: RegISize;
// SAFETY: calling getpid is safe.
unsafe {
- let n = sys::SYS_getpid;
- let r: i64;
+ #[cfg(target_arch = "aarch64")]
asm!(
"svc 0",
- in("x8") n as u64,
+ in("x8") n,
out("x0") r,
+ // Do not use `preserves_flags` because AArch64 FreeBSD syscalls modify the condition flags.
options(nostack, readonly),
);
- r as pid_t
- }
- }
- #[cfg(target_arch = "aarch64")]
- #[inline]
- unsafe fn sysctl(
- name: *const c_int,
- name_len: c_uint,
- old_p: *mut c_void,
- old_len_p: *mut c_size_t,
- new_p: *const c_void,
- new_len: c_size_t,
- ) -> Result<c_int, c_int> {
- #[allow(clippy::cast_possible_truncation)]
- // SAFETY: the caller must uphold the safety contract.
- unsafe {
- let mut n = sys::SYS___sysctl as u64;
- let r: i64;
- asm!(
- "svc 0",
- "b.cc 2f",
- "mov x8, x0",
- "mov x0, #-1",
- "2:",
- inout("x8") n,
- inout("x0") ptr_reg!(name) => r,
- inout("x1") name_len as u64 => _,
- in("x2") ptr_reg!(old_p),
- in("x3") ptr_reg!(old_len_p),
- in("x4") ptr_reg!(new_p),
- in("x5") new_len as u64,
- options(nostack),
- );
- if r as c_int == -1 {
- Err(n as c_int)
- } else {
- Ok(r as c_int)
- }
- }
- }
-
- // https://github.com/freebsd/freebsd-src/blob/release/14.1.0/lib/libc/powerpc64/SYS.h
- #[cfg(target_arch = "powerpc64")]
- #[inline]
- fn getpid() -> pid_t {
- #[allow(clippy::cast_possible_truncation)]
- // SAFETY: calling getpid is safe.
- unsafe {
- let n = sys::SYS_getpid;
- let r: i64;
+ #[cfg(target_arch = "powerpc64")]
asm!(
"sc",
- inout("r0") n as u64 => _,
+ inout("r0") n => _,
out("r3") r,
out("r4") _,
out("r5") _,
@@ -1005,12 +1159,16 @@ mod tests {
out("r11") _,
out("r12") _,
out("cr0") _,
+ out("ctr") _,
+ out("xer") _,
options(nostack, preserves_flags, readonly),
);
+ }
+ #[allow(clippy::cast_possible_truncation)]
+ {
r as pid_t
}
}
- #[cfg(target_arch = "powerpc64")]
#[inline]
unsafe fn sysctl(
name: *const c_int,
@@ -1020,37 +1178,59 @@ mod tests {
new_p: *const c_void,
new_len: c_size_t,
) -> Result<c_int, c_int> {
- #[allow(clippy::cast_possible_truncation)]
+ let mut n = sys::SYS___sysctl as RegSize;
+ let arg1 = ptr_reg!(name);
+ let arg2 = name_len as RegSize;
+ let arg3 = ptr_reg!(old_p);
+ let arg4 = ptr_reg!(old_len_p);
+ let arg5 = ptr_reg!(new_p);
+ let arg6 = new_len as RegSize;
+ let r: RegISize;
// SAFETY: the caller must uphold the safety contract.
unsafe {
- let mut n = sys::SYS___sysctl as u64;
- let r: i64;
+ #[cfg(target_arch = "aarch64")]
+ asm!(
+ "svc 0",
+ "b.cc 2f",
+ "mov x8, x0",
+ "mov x0, #-1",
+ "2:",
+ inout("x8") n,
+ inout("x0") arg1 => r,
+ inout("x1") arg2 => _,
+ in("x2") arg3,
+ in("x3") arg4,
+ in("x4") arg5,
+ in("x5") arg6,
+ // Do not use `preserves_flags` because AArch64 FreeBSD syscalls modify the condition flags.
+ options(nostack),
+ );
+ #[cfg(target_arch = "powerpc64")]
asm!(
"sc",
"bns+ 2f",
"mr %r0, %r3",
"li %r3, -1",
"2:",
inout("r0") n,
- inout("r3") ptr_reg!(name) => r,
- inout("r4") name_len as u64 => _,
- inout("r5") ptr_reg!(old_p) => _,
- inout("r6") ptr_reg!(old_len_p) => _,
- inout("r7") ptr_reg!(new_p) => _,
- inout("r8") new_len as u64 => _,
+ inout("r3") arg1 => r,
+ inout("r4") arg2 => _,
+ inout("r5") arg3 => _,
+ inout("r6") arg4 => _,
+ inout("r7") arg5 => _,
+ inout("r8") arg6 => _,
out("r9") _,
out("r10") _,
out("r11") _,
out("r12") _,
out("cr0") _,
+ out("ctr") _,
+ out("xer") _,
options(nostack, preserves_flags),
);
- if r as c_int == -1 {
- Err(n as c_int)
- } else {
- Ok(r as c_int)
- }
}
+ #[allow(clippy::cast_possible_truncation)]
+ if r as c_int == -1 { Err(n as c_int) } else { Ok(r as c_int) }
}
let mut auxv: [sys::Elf_Auxinfo; sys::AT_COUNT as usize] = unsafe { mem::zeroed() };
@@ -1082,6 +1262,7 @@ mod tests {
}
for aux in &auxv {
+ #[allow(clippy::cast_sign_loss)]
if aux.a_type == type_ as c_long {
// SAFETY: aux.a_un is #[repr(C)] union and all fields have
// the same size and can be safely transmuted to integers.
@@ -1093,15 +1274,15 @@ mod tests {
// AT_HWCAP2 is only available on FreeBSD 13+ on AArch64.
let hwcap2_else = |e| if cfg!(target_arch = "aarch64") { 0 } else { panic!("{:?}", e) };
- assert_eq!(os::getauxval(ffi::AT_HWCAP), getauxval_sysctl_libc(ffi::AT_HWCAP).unwrap());
- assert_eq!(
- os::getauxval(ffi::AT_HWCAP2),
- getauxval_sysctl_libc(ffi::AT_HWCAP2).unwrap_or_else(hwcap2_else)
- );
- assert_eq!(os::getauxval(ffi::AT_HWCAP), getauxval_sysctl_no_libc(ffi::AT_HWCAP).unwrap());
- assert_eq!(
- os::getauxval(ffi::AT_HWCAP2),
- getauxval_sysctl_no_libc(ffi::AT_HWCAP2).unwrap_or_else(hwcap2_else)
- );
+ let at = ffi::AT_HWCAP;
+ assert_eq!(os::getauxval(at), getauxval_sysctl_libc(at).unwrap());
+ assert_eq!(os::getauxval(at), getauxval_sysctl_no_libc(at).unwrap());
+ let at = ffi::AT_HWCAP2;
+ assert_eq!(os::getauxval(at), getauxval_sysctl_libc(at).unwrap_or_else(hwcap2_else));
+ assert_eq!(os::getauxval(at), getauxval_sysctl_no_libc(at).unwrap_or_else(hwcap2_else));
+ for &at in &[ffi::AT_HWCAP3, ffi::AT_HWCAP4] {
+ assert_eq!(os::getauxval(at), getauxval_sysctl_libc(at).unwrap_or_default());
+ assert_eq!(os::getauxval(at), getauxval_sysctl_no_libc(at).unwrap_or_default());
+ }
}
}
### external/vendor/portable-atomic/src/imp/detect/common.rs
@@ -6,20 +6,23 @@ pub(crate) struct CpuInfo(u32);
impl CpuInfo {
#[inline]
- fn set(&mut self, bit: u32) {
- self.0 = set(self.0, bit);
+ fn set(&mut self, bit: CpuInfoFlag) {
+ self.0 = set(self.0, bit as u32);
}
#[inline]
- fn test(self, bit: u32) -> bool {
- test(self.0, bit)
+ #[must_use]
+ fn test(self, bit: CpuInfoFlag) -> bool {
+ test(self.0, bit as u32)
}
}
#[inline]
+#[must_use]
fn set(x: u32, bit: u32) -> u32 {
- x | 1 << bit
+ x | (1 << bit)
}
#[inline]
+#[must_use]
fn test(x: u32, bit: u32) -> bool {
x & (1 << bit) != 0
}
@@ -33,11 +36,10 @@ pub(crate) fn detect() -> CpuInfo {
if info.0 != 0 {
return info;
}
- info.set(CpuInfo::INIT);
- // Note: detect_false cfg is intended to make it easy for portable-atomic developers to
- // test cases such as has_cmpxchg16b == false, has_lse == false,
- // __kuser_helper_version < 5, etc., and is not a public API.
- if !cfg!(portable_atomic_test_outline_atomics_detect_false) {
+ info.set(CpuInfoFlag::Init);
+ // Note: detect_false cfg is intended to make it easy for developers to test
+ // cases where features usually available is not available, and is not a public API.
+ if !cfg!(portable_atomic_test_detect_false) {
_detect(&mut info);
}
CACHE.store(info.0, Ordering::Relaxed);
@@ -47,280 +49,150 @@ pub(crate) fn detect() -> CpuInfo {
macro_rules! flags {
($(
$(#[$attr:meta])*
- $flag:ident ($shift:literal, $func:ident, $name:literal, $cfg:meta),
+ $func:ident($name:literal, any($($cfg:ident),*)),
)*) => {
+ #[allow(dead_code, non_camel_case_types)]
+ #[derive(Clone, Copy)]
+ #[cfg_attr(test, derive(PartialEq, Eq, PartialOrd, Ord))]
+ #[repr(u32)]
+ enum CpuInfoFlag {
+ Init = 0,
+ $($func,)*
+ }
impl CpuInfo {
- const INIT: u32 = 0;
$(
$(#[$attr])*
- const $flag: u32 = $shift;
- $(#[$attr])*
- #[cfg(any(test, not($cfg)))]
+ #[cfg(any(test, not(any($($cfg = $name),*))))]
#[inline]
+ #[must_use]
pub(crate) fn $func(self) -> bool {
- self.test(Self::$flag)
+ self.test(CpuInfoFlag::$func)
}
)*
#[cfg(test)] // for test
- const ALL_FLAGS: &'static [(&'static str, u32, bool)] = &[$(
- ($name, Self::$flag, cfg!($cfg)),
+ const ALL_FLAGS: &'static [(&'static str, CpuInfoFlag, bool)] = &[$(
+ ($name, CpuInfoFlag::$func, cfg!(any($($cfg = $name),*))),
)*];
}
+ #[test]
+ #[cfg_attr(portable_atomic_test_detect_false, ignore = "detection disabled")]
+ fn test_detect() {$(
+ $(#[$attr])*
+ {
+ const _: u32 = 1_u32 << CpuInfoFlag::$func as u32;
+ assert_eq!($name.replace(|c: char| c == '-' || c == '.', "_"), stringify!($func));
+ if detect().$func() {
+ assert!(detect().test(CpuInfoFlag::$func));
+ } else {
+ assert!(!detect().test(CpuInfoFlag::$func));
+ }
+ }
+ )*}
};
}
+// rustc definitions: https://github.com/rust-lang/rust/blob/ddaf12390d3ffb7d5ba74491a48f3cd528e5d777/compiler/rustc_target/src/target_features.rs
+
+// LLVM definitions: https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/AArch64/AArch64Features.td
#[cfg(any(target_arch = "aarch64", target_arch = "arm64ec"))]
flags! {
+ // The Armv8.1 architecture extension
+ // https://developer.arm.com/documentation/109697/2025_06/Feature-descriptions/The-Armv8-1-architecture-extension
// FEAT_LSE, Large System Extensions
- // https://developer.arm.com/documentation/109697/0100/Feature-descriptions/The-Armv8-1-architecture-extension
// > This feature is supported in AArch64 state only.
// > FEAT_LSE is OPTIONAL from Armv8.0.
// > FEAT_LSE is mandatory from Armv8.1.
- HAS_LSE(1, has_lse, "lse", any(target_feature = "lse", portable_atomic_target_feature = "lse")),
+ lse("lse", any(target_feature /* 1.61+ */, portable_atomic_target_feature)),
+
+ // The Armv8.3 architecture extension
+ // https://developer.arm.com/documentation/109697/2025_06/Feature-descriptions/The-Armv8-3-architecture-extension
+ // FEAT_LRCPC, Load-Acquire RCpc instructions
+ // > These instructions are added to the A64 instruction set only.
+ // > FEAT_LRCPC is OPTIONAL from Armv8.2.
+ // > FEAT_LRCPC is mandatory from Armv8.3.
+ #[cfg(test)] // test-only
+ rcpc("rcpc", any(target_feature /* 1.61+ */)),
+
+ // The Armv8.4 architecture extension
+ // https://developer.arm.com/documentation/109697/2025_06/Feature-descriptions/The-Armv8-4-architecture-extension
// FEAT_LSE2, Large System Extensions version 2
- // https://developer.arm.com/documentation/109697/0100/Feature-descriptions/The-Armv8-4-architecture-extension
// > This feature is supported in AArch64 state only.
// > FEAT_LSE2 is OPTIONAL from Armv8.2.
// > FEAT_LSE2 is mandatory from Armv8.4.
#[cfg_attr(not(test), allow(dead_code))]
- HAS_LSE2(2, has_lse2, "lse2", any(target_feature = "lse2", portable_atomic_target_feature = "lse2")),
+ lse2("lse2", any(target_feature /* nightly */, portable_atomic_target_feature)),
+ // FEAT_LRCPC2, Load-Acquire RCpc instructions version 2
+ // > These instructions are added to the A64 instruction set only.
+ // > FEAT_LRCPC2 is OPTIONAL from Armv8.2.
+ // > FEAT_LRCPC2 is mandatory from Armv8.4.
+ // > If FEAT_LRCPC2 is implemented, then FEAT_LRCPC is implemented.
+ #[cfg(test)] // test-only
+ rcpc2("rcpc2", any(target_feature /* 1.61+ */)),
+
+ // The Armv8.9 architecture extension
+ // https://developer.arm.com/documentation/109697/2025_06/Feature-descriptions/The-Armv8-9-architecture-extension
// FEAT_LRCPC3, Load-Acquire RCpc instructions version 3
- // https://developer.arm.com/documentation/109697/0100/Feature-descriptions/The-Armv8-9-architecture-extension
// > This feature is supported in AArch64 state only.
// > FEAT_LRCPC3 is OPTIONAL from Armv8.2.
// > If FEAT_LRCPC3 is implemented, then FEAT_LRCPC2 is implemented.
#[cfg_attr(not(test), allow(dead_code))]
- HAS_RCPC3(3, has_rcpc3, "rcpc3", any(target_feature = "rcpc3", portable_atomic_target_feature = "rcpc3")),
+ rcpc3("rcpc3", any(target_feature /* nightly */, portable_atomic_target_feature)),
+
+ // The Armv9.4 architecture extension
+ // https://developer.arm.com/documentation/109697/2025_06/Feature-descriptions/The-Armv9-4-architecture-extension
// FEAT_LSE128, 128-bit Atomics
- // https://developer.arm.com/documentation/109697/0100/Feature-descriptions/The-Armv9-4-architecture-extension
// > This feature is supported in AArch64 state only.
// > FEAT_LSE128 is OPTIONAL from Armv9.3.
// > If FEAT_LSE128 is implemented, then FEAT_LSE is implemented.
#[cfg_attr(not(test), allow(dead_code))]
- HAS_LSE128(4, has_lse128, "lse128", any(target_feature = "lse128", portable_atomic_target_feature = "lse128")),
+ lse128("lse128", any(target_feature /* nightly */, portable_atomic_target_feature)),
+
+ // The Armv9.6 architecture extension
+ // https://developer.arm.com/documentation/109697/2025_06/Feature-descriptions/The-Armv9-6-architecture-extension
+ // FEAT_LSFE, Large System Float Extension
+ // > This feature is supported in AArch64 state only.
+ // > FEAT_LSFE is OPTIONAL from Armv9.3.
+ // > If FEAT_LSFE is implemented, then FEAT_FP is implemented.
+ #[cfg(test)] // test-only
+ lsfe("lsfe", any(target_feature /* N/A */, portable_atomic_target_feature)),
+
+ #[cfg(test)] // test-only
+ cpuid("cpuid", any(/* no corresponding target feature */)),
}
+// LLVM definitions: https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/ARM/ARMFeatures.td
+#[cfg(target_arch = "arm")]
+flags! {
+ #[cfg(test)] // test-only
+ lpae("lpae", any(/* no corresponding target feature */)),
+}
+
+// LLVM definitions: https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/PowerPC/PPC.td
#[cfg(target_arch = "powerpc64")]
flags! {
// lqarx and stqcx.
- HAS_QUADWORD_ATOMICS(1, has_quadword_atomics, "quadword-atomics", any(target_feature = "quadword-atomics", portable_atomic_target_feature = "quadword-atomics")),
+ quadword_atomics("quadword-atomics", any(target_feature /* nightly */, portable_atomic_target_feature)),
}
+// LLVM definitions: https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/RISCV/RISCVFeatures.td
#[cfg(any(target_arch = "riscv32", target_arch = "riscv64"))]
flags! {
// amocas.{w,d,q}
- HAS_ZACAS(1, has_zacas, "zacas", any(target_feature = "experimental-zacas", portable_atomic_target_feature = "experimental-zacas")),
+ zacas("zacas", any(target_feature /* 1.94+ */, portable_atomic_target_feature)),
+ #[cfg(test)] // test-only
+ zabha("zabha", any(target_feature /* 1.94+ */, portable_atomic_target_feature)),
+ #[cfg(test)] // test-only
+ zalasr("zalasr", any(/* no corresponding target feature */)),
}
+// LLVM definitions: https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/X86/X86.td
#[cfg(target_arch = "x86_64")]
flags! {
- // cmpxchg16b
- HAS_CMPXCHG16B(1, has_cmpxchg16b, "cmpxchg16b", any(target_feature = "cmpxchg16b", portable_atomic_target_feature = "cmpxchg16b")),
- // atomic vmovdqa
+ // avx
#[cfg(target_feature = "sse")]
- HAS_VMOVDQA_ATOMIC(2, has_vmovdqa_atomic, "vmovdqa-atomic", any(/* always false */)),
-}
-
-// core::ffi::c_* (except c_void) requires Rust 1.64, libc requires Rust 1.63
-#[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
-#[cfg(not(windows))]
-#[allow(dead_code, unused_macros, non_camel_case_types)]
-#[macro_use]
-mod c_types {
- /// Defines constants with #[cfg(test)] static assertions which checks
- /// values are the same as the platform's latest header files' ones.
- // Note: This macro is sys_const!({ }), not sys_const! { }.
- // An extra brace is used in input to make contents rustfmt-able:.
- macro_rules! sys_const {
- ({$(
- $(#[$attr:meta])*
- $vis:vis const $name:ident: $ty:ty = $val:expr;
- )*}) => {
- $(
- $(#[$attr])*
- $vis const $name: $ty = $val;
- )*
- // Static assertions for FFI bindings.
- // This checks that FFI bindings defined in this crate and FFI bindings generated for
- // the platform's latest header file using bindgen have the same values.
- // Since this is static assertion, we can detect problems with
- // `cargo check --tests --target <target>` run in CI (via TESTS=1 build.sh)
- // without actually running tests on these platforms.
- // See also https://github.com/taiki-e/test-helper/blob/HEAD/tools/codegen/src/ffi.rs.
- #[cfg(test)]
- #[allow(
- unused_attributes, // for #[allow(..)] in $(#[$attr])*
- clippy::cast_possible_wrap,
- clippy::cast_sign_loss,
- clippy::cast_possible_truncation,
- )]
- const _: fn() = || {$(
- $(#[$attr])*
- sys_const_cmp!($name, $ty);
- )*};
- };
- }
- #[cfg(test)]
- macro_rules! sys_const_cmp {
- (RTLD_DEFAULT, $ty:ty) => {
- // ptr comparison and ptr-to-int cast are not stable on const context, so use ptr-to-int
- // transmute and compare its result.
- static_assert!(
- // SAFETY: Pointer-to-integer transmutes are valid (since we are okay with losing the
- // provenance here). (Same as <pointer>::addr().)
- unsafe {
- core::mem::transmute::<$ty, usize>(RTLD_DEFAULT)
- == core::mem::transmute::<$ty, usize>(test_helper::sys::RTLD_DEFAULT)
- }
- );
- };
- ($name:ident, $ty:ty) => {
- static_assert!($name == test_helper::sys::$name as $ty);
- };
- }
- /// Defines functions with #[cfg(test)] static assertions which checks
- /// signatures are the same as the platform's latest header files' ones.
- // Note: This macro is sys_fn!({ }), not sys_fn! { }.
- // An extra brace is used in input to make contents rustfmt-able:.
- macro_rules! sys_fn {
- ({
- $(#[$extern_attr:meta])*
- extern $abi:literal {$(
- $(#[$fn_attr:meta])*
- $vis:vis fn $name:ident($($arg_pat:ident: $arg_ty:ty),* $(,)?) $(-> $ret_ty:ty)?;
- )*}
- }) => {
- $(#[$extern_attr])*
- extern $abi {$(
- $(#[$fn_attr])*
- $vis fn $name($($arg_pat: $arg_ty),*) $(-> $ret_ty)?;
- )*}
- // Static assertions for FFI bindings.
- // This checks that FFI bindings defined in this crate and FFI bindings generated for
- // the platform's latest header file using bindgen have the same signatures.
- // Since this is static assertion, we can detect problems with
- // `cargo check --tests --target <target>` run in CI (via TESTS=1 build.sh)
- // without actually running tests on these platforms.
- // See also https://github.com/taiki-e/test-helper/blob/HEAD/tools/codegen/src/ffi.rs.
- #[cfg(test)]
- const _: fn() = || {$(
- $(#[$fn_attr])*
- {
- let mut _f: unsafe extern $abi fn($($arg_ty),*) $(-> $ret_ty)? = $name;
- _f = test_helper::sys::$name;
- }
- )*};
- };
- }
- /// Defines #[repr(C)] structs with #[cfg(test)] static assertions which checks
- /// fields are the same as the platform's latest header files' ones.
- // Note: This macro is sys_struct!({ }), not sys_struct! { }.
- // An extra brace is used in input to make contents rustfmt-able:.
- macro_rules! sys_struct {
- ({$(
- $(#[$struct_attr:meta])*
- $struct_vis:vis struct $struct_name:ident {$(
- $(#[$field_attr:meta])*
- $field_vis:vis $field_name:ident: $field_ty:ty,
- )*}
- )*}) => {
- $(
- $(#[$struct_attr])*
- #[derive(Copy, Clone)]
- #[cfg_attr(test, derive(Debug, PartialEq))]
- #[repr(C)]
- $struct_vis struct $struct_name {$(
- $(#[$field_attr])*
- $field_vis $field_name: $field_ty,
- )*}
- )*
- // Static assertions for FFI bindings.
- // This checks that FFI bindings defined in this crate and FFI bindings generated for
- // the platform's latest header file using bindgen have the same fields.
- // Since this is static assertion, we can detect problems with
- // `cargo check --tests --target <target>` run in CI (via TESTS=1 build.sh)
- // without actually running tests on these platforms.
- // See also https://github.com/taiki-e/test-helper/blob/HEAD/tools/codegen/src/ffi.rs.
- #[cfg(test)]
- #[allow(clippy::undocumented_unsafe_blocks)]
- const _: fn() = || {$(
- $(#[$struct_attr])*
- {
- static_assert!(
- core::mem::size_of::<$struct_name>()
- == core::mem::size_of::<test_helper::sys::$struct_name>()
- );
- let s: $struct_name = unsafe { core::mem::zeroed() };
- // field names and types
- let _ = test_helper::sys::$struct_name {$(
- $(#[$field_attr])*
- $field_name: s.$field_name,
- )*};
- // field offsets
- #[cfg(not(portable_atomic_no_offset_of))]
- {$(
- $(#[$field_attr])*
- static_assert!(
- core::mem::offset_of!($struct_name, $field_name) ==
- core::mem::offset_of!(test_helper::sys::$struct_name, $field_name),
- );
- )*}
- }
- )*};
- };
- }
-
- pub(crate) type c_void = core::ffi::c_void;
- // c_{,u}int is {i,u}32 on non-16-bit architectures
- // https://github.com/rust-lang/rust/blob/1.80.0/library/core/src/ffi/mod.rs#L147
- // (16-bit architectures currently don't use this module)
- pub(crate) type c_int = i32;
- pub(crate) type c_uint = u32;
- // c_{,u}long is {i,u}64 on non-Windows 64-bit targets, otherwise is {i,u}32
- // https://github.com/rust-lang/rust/blob/1.80.0/library/core/src/ffi/mod.rs#L159
- // (Windows currently doesn't use this module - this module is cfg(not(windows)))
- #[cfg(target_pointer_width = "64")]
- pub(crate) type c_long = i64;
- #[cfg(target_pointer_width = "64")]
- pub(crate) type c_ulong = u64;
- #[cfg(not(target_pointer_width = "64"))]
- pub(crate) type c_long = i32;
- #[cfg(not(target_pointer_width = "64"))]
- pub(crate) type c_ulong = u32;
- // c_size_t is currently always usize
- // https://github.com/rust-lang/rust/blob/1.80.0/library/core/src/ffi/mod.rs#L67
- pub(crate) type c_size_t = usize;
- // c_char is u8 by default on most non-Apple/non-Windows Arm/PowerPC/RISC-V/s390x/Hexagon targets
- // (Linux/Android/FreeBSD/NetBSD/OpenBSD/VxWorks/Fuchsia/QNX Neutrino/Horizon/AIX/z/OS)
- // https://github.com/rust-lang/rust/blob/1.80.0/library/core/src/ffi/mod.rs#L83
- // https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/lldb/source/Utility/ArchSpec.cpp#L712
- // RISC-V https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/draft-20240829-13bfa9f54634cb60d86b9b333e109f077805b4b3/riscv-cc.adoc#cc-type-representations
- // Hexagon https://lists.llvm.org/pipermail/llvm-dev/attachments/20190916/21516a52/attachment-0001.pdf
- // AIX https://www.ibm.com/docs/en/xl-c-aix/13.1.3?topic=specifiers-character-types
- // z/OS https://www.ibm.com/docs/en/zos/3.1.0?topic=specifiers-character-types
- // (Windows currently doesn't use this module)
- #[cfg(not(target_vendor = "apple"))]
- pub(crate) type c_char = u8;
- // c_char is i8 on all Apple targets
- #[cfg(target_vendor = "apple")]
- pub(crate) type c_char = i8;
-
- // Static assertions for C type definitions.
- #[cfg(test)]
- const _: fn() = || {
- use test_helper::sys;
- let _: c_int = 0 as std::os::raw::c_int;
- let _: c_uint = 0 as std::os::raw::c_uint;
- let _: c_long = 0 as std::os::raw::c_long;
- let _: c_ulong = 0 as std::os::raw::c_ulong;
- let _: c_size_t = 0 as libc::size_t; // std::os::raw::c_size_t is unstable
- #[cfg(not(any(
- all(target_arch = "aarch64", target_os = "illumos"), // TODO: https://github.com/rust-lang/rust/issues/129945
- all(target_arch = "riscv64", target_os = "android"), // TODO: https://github.com/rust-lang/rust/issues/129945
- )))]
- let _: c_char = 0 as std::os::raw::c_char;
- let _: c_char = 0 as sys::c_char;
- };
+ avx("avx", any(target_feature)),
+ // cmpxchg16b
+ cmpxchg16b("cmpxchg16b", any(target_feature /* 1.69+ */, portable_atomic_target_feature)),
}
#[allow(
@@ -338,7 +210,7 @@ mod tests_common {
#[test]
fn test_bit_flags() {
- let mut flags = vec![("init", CpuInfo::INIT)];
+ let mut flags = vec![("init", CpuInfoFlag::Init)];
flags.extend(CpuInfo::ALL_FLAGS.iter().map(|&(name, flag, _)| (name, flag)));
let flag_set = flags.iter().map(|(_, flag)| flag).collect::<BTreeSet<_>>();
let name_set = flags.iter().map(|(_, flag)| flag).collect::<BTreeSet<_>>();
@@ -369,11 +241,7 @@ mod tests_common {
#[test]
fn print_features() {
- use std::{
- fmt::Write as _,
- io::{self, Write},
- string::String,
- };
+ use std::{fmt::Write as _, string::String};
let mut features = String::new();
features.push_str("\nfeatures:\n");
@@ -389,106 +257,14 @@ mod tests_common {
);
}
}
- let stdout = io::stderr();
- let mut stdout = stdout.lock();
- let _ = stdout.write_all(features.as_bytes());
+ test_helper::eprintln_nocapture!("{}", features);
}
- #[cfg(any(target_arch = "aarch64", target_arch = "arm64ec"))]
- #[test]
- #[cfg_attr(portable_atomic_test_outline_atomics_detect_false, ignore)]
- fn test_detect() {
- let proc_cpuinfo = test_helper::cpuinfo::ProcCpuinfo::new();
- if detect().has_lse() {
- assert!(detect().test(CpuInfo::HAS_LSE));
- if let Ok(proc_cpuinfo) = proc_cpuinfo {
- assert!(proc_cpuinfo.lse);
- }
- } else {
- assert!(!detect().test(CpuInfo::HAS_LSE));
- if let Ok(proc_cpuinfo) = proc_cpuinfo {
- assert!(!proc_cpuinfo.lse);
- }
- }
- if detect().has_lse2() {
- assert!(detect().test(CpuInfo::HAS_LSE));
- assert!(detect().test(CpuInfo::HAS_LSE2));
- if let Ok(test_helper::cpuinfo::ProcCpuinfo { lse2: Some(lse2), .. }) = proc_cpuinfo {
- assert!(lse2);
- }
- } else {
- assert!(!detect().test(CpuInfo::HAS_LSE2));
- if let Ok(test_helper::cpuinfo::ProcCpuinfo { lse2: Some(lse2), .. }) = proc_cpuinfo {
- assert!(!lse2);
- }
- }
- if detect().has_lse128() {
- assert!(detect().test(CpuInfo::HAS_LSE));
- assert!(detect().test(CpuInfo::HAS_LSE2));
- assert!(detect().test(CpuInfo::HAS_LSE128));
- if let Ok(test_helper::cpuinfo::ProcCpuinfo { lse128: Some(lse128), .. }) = proc_cpuinfo
- {
- assert!(lse128);
- }
- } else {
- assert!(!detect().test(CpuInfo::HAS_LSE128));
- if let Ok(test_helper::cpuinfo::ProcCpuinfo { lse128: Some(lse128), .. }) = proc_cpuinfo
- {
- assert!(!lse128);
- }
- }
- if detect().has_rcpc3() {
- assert!(detect().test(CpuInfo::HAS_RCPC3));
- if let Ok(test_helper::cpuinfo::ProcCpuinfo { rcpc3: Some(rcpc3), .. }) = proc_cpuinfo {
- assert!(rcpc3);
- }
- } else {
- assert!(!detect().test(CpuInfo::HAS_RCPC3));
- if let Ok(test_helper::cpuinfo::ProcCpuinfo { rcpc3: Some(rcpc3), .. }) = proc_cpuinfo {
- assert!(!rcpc3);
- }
- }
- }
- #[cfg(target_arch = "powerpc64")]
- #[test]
- #[cfg_attr(portable_atomic_test_outline_atomics_detect_false, ignore)]
- fn test_detect() {
- let proc_cpuinfo = test_helper::cpuinfo::ProcCpuinfo::new();
- if detect().has_quadword_atomics() {
- assert!(detect().test(CpuInfo::HAS_QUADWORD_ATOMICS));
- if let Ok(proc_cpuinfo) = proc_cpuinfo {
- assert!(proc_cpuinfo.power8);
- }
- } else {
- assert!(!detect().test(CpuInfo::HAS_QUADWORD_ATOMICS));
- if let Ok(proc_cpuinfo) = proc_cpuinfo {
- assert!(!proc_cpuinfo.power8);
- }
- }
- }
- #[cfg(any(target_arch = "riscv32", target_arch = "riscv64"))]
- #[test]
- #[cfg_attr(portable_atomic_test_outline_atomics_detect_false, ignore)]
- fn test_detect() {
- if detect().has_zacas() {
- assert!(detect().test(CpuInfo::HAS_ZACAS));
- } else {
- assert!(!detect().test(CpuInfo::HAS_ZACAS));
- }
- }
- #[cfg(target_arch = "x86_64")]
- #[test]
- #[cfg_attr(portable_atomic_test_outline_atomics_detect_false, ignore)]
- fn test_detect() {
- if detect().has_cmpxchg16b() {
- assert!(detect().test(CpuInfo::HAS_CMPXCHG16B));
- } else {
- assert!(!detect().test(CpuInfo::HAS_CMPXCHG16B));
- }
- if detect().has_vmovdqa_atomic() {
- assert!(detect().test(CpuInfo::HAS_VMOVDQA_ATOMIC));
- } else {
- assert!(!detect().test(CpuInfo::HAS_VMOVDQA_ATOMIC));
- }
- }
+ // Static assertions for C type definitions.
+ // Assertions with core::ffi types are in crate::utils::ffi module.
+ #[cfg(not(any(windows, target_arch = "x86", target_arch = "x86_64")))]
+ const _: fn() = || {
+ use test_helper::sys;
+ let _: crate::utils::ffi::c_char = 0 as sys::c_char;
+ };
}
### external/vendor/portable-atomic/src/imp/detect/powerpc64_aix.rs
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: Apache-2.0 OR MIT
+
+/*
+Run-time CPU feature detection on PowerPC64 AIX by using getsystemcfg.
+
+Refs:
+- https://github.com/golang/go/blob/go1.25.0/src/internal/cpu/cpu_ppc64x_aix.go
+
+As of nightly-2024-09-07, is_powerpc_feature_detected doesn't support run-time detection on AIX.
+https://github.com/rust-lang/stdarch/blob/d9466edb4c53cece8686ee6e17b028436ddf4151/crates/std_detect/src/detect/mod.rs
+
+Run-time detection on PowerPC64 AIX is currently disabled by default as experimental
+because we cannot run tests on the VM or real machine.
+*/
+
+include!("common.rs");
+
+// libc requires Rust 1.63
+mod ffi {
+ pub(crate) use crate::utils::ffi::{c_int, c_ulong};
+
+ sys_const!({
+ // https://github.com/rust-lang/libc/blob/0.2.158/src/unix/aix/mod.rs#L2058
+ // https://github.com/golang/go/blob/go1.25.0/src/internal/cpu/cpu_ppc64x_aix.go
+ pub(crate) const SC_IMPL: c_int = 2;
+ pub(crate) const POWER_8: c_ulong = 0x10000;
+ pub(crate) const POWER_9: c_ulong = 0x20000;
+ });
+ // TODO: use sys_const! once libc crate defined it.
+ pub(crate) const POWER_10: c_ulong = 0x40000;
+
+ sys_fn!({
+ extern "C" {
+ // https://www.ibm.com/docs/en/aix/7.3?topic=g-getsystemcfg-subroutine
+ // https://github.com/rust-lang/libc/blob/0.2.158/src/unix/aix/powerpc64.rs#L643
+ pub(crate) fn getsystemcfg(name: c_int) -> c_ulong;
+ }
+ });
+}
+
+#[cold]
+fn _detect(info: &mut CpuInfo) {
+ // SAFETY: calling getsystemcfg is safe.
+ let impl_ = unsafe { ffi::getsystemcfg(ffi::SC_IMPL) };
+ if impl_ == ffi::c_ulong::MAX {
+ return;
+ }
+ // Check both POWER_8 and later ISAs (which are superset of POWER_8) because
+ // AIX currently doesn't set POWER_8 when POWER_9 is set.
+ // https://github.com/golang/go/commit/51859ec2292d9c1d82a7054ec672ff551a0d7497
+ if impl_ & (ffi::POWER_8 | ffi::POWER_9 | ffi::POWER_10) != 0 {
+ info.set(CpuInfoFlag::quadword_atomics);
+ }
+}
### external/vendor/portable-atomic/src/imp/detect/riscv_linux.rs
@@ -6,20 +6,20 @@ Run-time CPU feature detection on RISC-V Linux/Android by using riscv_hwprobe.
On RISC-V, detection using auxv only supports single-letter extensions.
So, we use riscv_hwprobe that supports multi-letter extensions.
-Refs: https://github.com/torvalds/linux/blob/v6.11/Documentation/arch/riscv/hwprobe.rst
+Refs: https://github.com/torvalds/linux/blob/v6.16/Documentation/arch/riscv/hwprobe.rst
*/
include!("common.rs");
use core::ptr;
-// core::ffi::c_* (except c_void) requires Rust 1.64, libc requires Rust 1.63
+// libc requires Rust 1.63
#[allow(non_camel_case_types, non_upper_case_globals)]
mod ffi {
- pub(crate) use super::c_types::{c_long, c_size_t, c_uint, c_ulong};
+ pub(crate) use crate::utils::ffi::{c_long, c_size_t, c_uint, c_ulong};
sys_struct!({
- // https://github.com/torvalds/linux/blob/v6.11/arch/riscv/include/uapi/asm/hwprobe.h
+ // https://github.com/torvalds/linux/blob/v6.16/arch/riscv/include/uapi/asm/hwprobe.h
pub(crate) struct riscv_hwprobe {
pub(crate) key: i64,
pub(crate) value: u64,
@@ -29,59 +29,128 @@ mod ffi {
sys_const!({
pub(crate) const __NR_riscv_hwprobe: c_long = 258;
- // https://github.com/torvalds/linux/blob/v6.11/arch/riscv/include/uapi/asm/hwprobe.h
+ // https://github.com/torvalds/linux/blob/v6.16/arch/riscv/include/uapi/asm/hwprobe.h
+ // Linux 6.4+
+ // https://github.com/torvalds/linux/commit/00e76e2c6a2bd3976d44d4a1fdd0b7a3c2566607
+ pub(crate) const RISCV_HWPROBE_KEY_BASE_BEHAVIOR: i64 = 3;
+ pub(crate) const RISCV_HWPROBE_BASE_BEHAVIOR_IMA: u64 = 1 << 0;
pub(crate) const RISCV_HWPROBE_KEY_IMA_EXT_0: i64 = 4;
// Linux 6.8+
// https://github.com/torvalds/linux/commit/154a3706122978eeb34d8223d49285ed4f3c61fa
pub(crate) const RISCV_HWPROBE_EXT_ZACAS: u64 = 1 << 34;
+ // Linux 6.16+
+ // https://github.com/torvalds/linux/commit/415a8c81da3dab0a585bd4f8d505a11ad5a171a7
+ #[cfg(test)]
+ pub(crate) const RISCV_HWPROBE_EXT_ZABHA: u64 = 1 << 58;
+ // Linux 6.19+
+ // https://github.com/torvalds/linux/commit/f4922b69165735e81752ee47d174f873e989a449
+ #[cfg(test)]
+ pub(crate) const RISCV_HWPROBE_EXT_ZALASR: u64 = 1 << 59;
});
- // TODO: use sys_fn!
- #[cfg(not(all(
- target_os = "linux",
- any(target_arch = "riscv32", all(target_arch = "riscv64", target_pointer_width = "64")),
- )))]
- extern "C" {
- // https://man7.org/linux/man-pages/man2/syscall.2.html
- pub(crate) fn syscall(number: c_long, ...) -> c_long;
- }
- // Use asm-based syscall for compatibility with non-libc targets if possible.
- #[cfg(all(
- target_os = "linux", // https://github.com/bytecodealliance/rustix/issues/1095
- any(target_arch = "riscv32", all(target_arch = "riscv64", target_pointer_width = "64")),
- ))]
- #[inline]
- pub(crate) unsafe fn syscall(
- number: c_long,
- a0: *mut riscv_hwprobe,
- a1: c_size_t,
- a2: c_size_t,
- a3: *mut c_ulong,
- a4: c_uint,
- ) -> c_long {
- // arguments must be extended to 64-bit if RV64
- let a4 = a4 as usize;
- let r;
- // SAFETY: the caller must uphold the safety contract.
- // Refs:
- // - https://github.com/bminor/musl/blob/v1.2.5/arch/riscv32/syscall_arch.h
- // - https://github.com/bminor/musl/blob/v1.2.5/arch/riscv64/syscall_arch.h
- unsafe {
- core::arch::asm!(
- "ecall",
- in("a7") number,
- inout("a0") a0 => r,
- in("a1") a1,
- in("a2") a2,
- in("a3") a3,
- in("a4") a4,
- options(nostack, preserves_flags)
- );
+ cfg_sel!({
+ // Use asm-based syscall on Linux for compatibility with non-libc targets if possible.
+ // Do not use it on Android, see https://github.com/bytecodealliance/rustix/issues/1095 for details.
+ #[cfg(all(
+ target_os = "linux",
+ any(
+ target_arch = "riscv32",
+ all(target_arch = "riscv64", target_pointer_width = "64"),
+ ),
+ ))]
+ {
+ #[cfg(not(portable_atomic_no_asm))]
+ use core::arch::asm;
+
+ use crate::utils::{RegISize, RegSize};
+
+ // Refs:
+ // - https://github.com/bminor/musl/blob/v1.2.5/arch/riscv32/syscall_arch.h
+ // - https://github.com/bminor/musl/blob/v1.2.5/arch/riscv64/syscall_arch.h
+ #[inline]
+ pub(crate) unsafe fn syscall5(
+ number: c_long,
+ arg1: *mut riscv_hwprobe,
+ arg2: c_size_t,
+ arg3: c_size_t,
+ arg4: *mut c_ulong,
+ arg5: c_uint,
+ ) -> c_long {
+ // arguments must be extended to 64-bit if 64-bit arch
+ #[allow(clippy::cast_possible_truncation)]
+ let number = number as RegISize;
+ let arg1 = ptr_reg!(arg1);
+ let arg2 = arg2 as RegSize;
+ let arg3 = arg3 as RegSize;
+ let arg4 = ptr_reg!(arg4);
+ let arg5 = arg5 as RegSize;
+ let r: RegISize;
+ // SAFETY: the caller must uphold the safety contract.
+ unsafe {
+ asm!(
+ "ecall",
+ in("a7") number,
+ inout("a0") arg1 => r,
+ in("a1") arg2,
+ in("a2") arg3,
+ in("a3") arg4,
+ in("a4") arg5,
+ // Clobber vector registers and do not use `preserves_flags` because RISC-V Linux syscalls don't preserve them.
+ // https://github.com/torvalds/linux/blob/v6.18/Documentation/arch/riscv/vector.rst#3--vector-register-state-across-system-calls
+ out("v0") _,
+ out("v1") _,
+ out("v2") _,
+ out("v3") _,
+ out("v4") _,
+ out("v5") _,
+ out("v6") _,
+ out("v7") _,
+ out("v8") _,
+ out("v9") _,
+ out("v10") _,
+ out("v11") _,
+ out("v12") _,
+ out("v13") _,
+ out("v14") _,
+ out("v15") _,
+ out("v16") _,
+ out("v17") _,
+ out("v18") _,
+ out("v19") _,
+ out("v20") _,
+ out("v21") _,
+ out("v22") _,
+ out("v23") _,
+ out("v24") _,
+ out("v25") _,
+ out("v26") _,
+ out("v27") _,
+ out("v28") _,
+ out("v29") _,
+ out("v30") _,
+ out("v31") _,
+ options(nostack),
+ );
+ }
+ #[allow(clippy::cast_possible_truncation)]
+ {
+ r as c_long
+ }
+ }
}
- r
- }
+ #[cfg(else)]
+ {
+ sys_fn!({
+ extern "C" {
+ // https://man7.org/linux/man-pages/man2/syscall.2.html
+ pub(crate) fn syscall(number: c_long, ...) -> c_long;
+ }
+ });
+ pub(crate) use self::syscall as syscall5;
+ }
+ });
- // https://github.com/torvalds/linux/blob/v6.11/Documentation/arch/riscv/hwprobe.rst
+ // https://github.com/torvalds/linux/blob/v6.16/Documentation/arch/riscv/hwprobe.rst
pub(crate) unsafe fn __riscv_hwprobe(
pairs: *mut riscv_hwprobe,
pair_count: c_size_t,
@@ -90,26 +159,43 @@ mod ffi {
flags: c_uint,
) -> c_long {
// SAFETY: the caller must uphold the safety contract.
- unsafe { syscall(__NR_riscv_hwprobe, pairs, pair_count, cpu_set_size, cpus, flags) }
+ unsafe { syscall5(__NR_riscv_hwprobe, pairs, pair_count, cpu_set_size, cpus, flags) }
}
}
// syscall returns an unsupported error if riscv_hwprobe is not supported,
// so we can safely use this function on older versions of Linux.
-fn riscv_hwprobe(out: &mut ffi::riscv_hwprobe) -> bool {
+fn riscv_hwprobe(out: &mut [ffi::riscv_hwprobe]) -> bool {
+ let len = out.len();
// SAFETY: We've passed the valid pointer and length,
// passing null ptr for cpus is safe because cpu_set_size is zero.
- unsafe { ffi::__riscv_hwprobe(out, 1, 0, ptr::null_mut(), 0) == 0 }
+ unsafe { ffi::__riscv_hwprobe(out.as_mut_ptr(), len, 0, ptr::null_mut(), 0) == 0 }
}
#[cold]
fn _detect(info: &mut CpuInfo) {
- let mut out = ffi::riscv_hwprobe { key: ffi::RISCV_HWPROBE_KEY_IMA_EXT_0, value: 0 };
- if riscv_hwprobe(&mut out) && out.key != -1 {
- let value = out.value;
- if value & ffi::RISCV_HWPROBE_EXT_ZACAS != 0 {
- info.set(CpuInfo::HAS_ZACAS);
+ let mut out = [
+ ffi::riscv_hwprobe { key: ffi::RISCV_HWPROBE_KEY_BASE_BEHAVIOR, value: 0 },
+ ffi::riscv_hwprobe { key: ffi::RISCV_HWPROBE_KEY_IMA_EXT_0, value: 0 },
+ ];
+ if riscv_hwprobe(&mut out)
+ && out[0].key != -1
+ && out[0].value & ffi::RISCV_HWPROBE_BASE_BEHAVIOR_IMA != 0
+ && out[1].key != -1
+ {
+ let value = out[1].value;
+ macro_rules! check {
+ ($flag:ident, $bit:ident) => {
+ if value & ffi::$bit != 0 {
+ info.set(CpuInfoFlag::$flag);
+ }
+ };
}
+ check!(zacas, RISCV_HWPROBE_EXT_ZACAS);
+ #[cfg(test)]
+ check!(zabha, RISCV_HWPROBE_EXT_ZABHA);
+ #[cfg(test)]
+ check!(zalasr, RISCV_HWPROBE_EXT_ZALASR);
}
}
@@ -140,40 +226,16 @@ mod tests {
libc::syscall(ffi::__NR_riscv_hwprobe, pairs, pair_count, cpu_set_size, cpus, flags)
}
}
- fn riscv_hwprobe_libc(out: &mut ffi::riscv_hwprobe) -> bool {
- unsafe { __riscv_hwprobe_libc(out, 1, 0, ptr::null_mut(), 0) == 0 }
+ fn riscv_hwprobe_libc(out: &mut [ffi::riscv_hwprobe]) -> bool {
+ let len = out.len();
+ unsafe { __riscv_hwprobe_libc(out.as_mut_ptr(), len, 0, ptr::null_mut(), 0) == 0 }
}
- let mut out = ffi::riscv_hwprobe { key: ffi::RISCV_HWPROBE_KEY_IMA_EXT_0, value: 0 };
- let mut libc_out = ffi::riscv_hwprobe { key: ffi::RISCV_HWPROBE_KEY_IMA_EXT_0, value: 0 };
+ let mut out = [
+ ffi::riscv_hwprobe { key: ffi::RISCV_HWPROBE_KEY_BASE_BEHAVIOR, value: 0 },
+ ffi::riscv_hwprobe { key: ffi::RISCV_HWPROBE_KEY_IMA_EXT_0, value: 0 },
+ ];
+ let mut libc_out = out;
assert_eq!(riscv_hwprobe(&mut out), riscv_hwprobe_libc(&mut libc_out));
assert_eq!(out, libc_out);
}
-
- // Static assertions for FFI bindings.
- // This checks that FFI bindings defined in this crate, FFI bindings defined
- // in libc, and FFI bindings generated for the platform's latest header file
- // using bindgen have compatible signatures.
- // Since this is static assertion, we can detect problems with
- // `cargo check --tests --target <target>` run in CI (via TESTS=1 build.sh)
- // without actually running tests on these platforms.
- // As for constants, they are checked by static assertions generated by sys_const!.
- // As for structs, they are checked by static assertions generated by sys_struct!.
- // See also https://github.com/taiki-e/test-helper/blob/HEAD/tools/codegen/src/ffi.rs.
- // TODO(codegen): auto-generate this test
- const _: fn() = || {
- #[cfg(not(all(
- target_os = "linux",
- any(
- target_arch = "riscv32",
- all(target_arch = "riscv64", target_pointer_width = "64"),
- ),
- )))]
- {
- use test_helper::sys;
- let mut _syscall: unsafe extern "C" fn(num: ffi::c_long, ...) -> ffi::c_long =
- ffi::syscall;
- _syscall = libc::syscall;
- _syscall = sys::syscall;
- }
- };
}
### external/vendor/portable-atomic/src/imp/detect/x86_64.rs
@@ -3,13 +3,13 @@
/*
Run-time CPU feature detection on x86_64 by using CPUID.
-Adapted from https://github.com/rust-lang/stdarch.
+Adapted from https://github.com/rust-lang/rust/blob/1.92.0/library/std_detect/src/detect/os/x86.rs.
*/
#![cfg_attr(portable_atomic_sanitize_thread, allow(dead_code))]
// Miri doesn't support inline assembly used in __cpuid: https://github.com/rust-lang/miri/issues/932
-// SGX doesn't support CPUID: https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/core_arch/src/x86/cpuid.rs#L102-L105
+// SGX doesn't support CPUID: https://github.com/rust-lang/rust/blob/1.92.0/library/std_detect/src/detect/os/x86.rs#L30-L33
#[cfg(any(target_env = "sgx", miri))]
compile_error!("internal error: this module is not supported on this environment");
@@ -32,15 +32,15 @@ fn __cpuid(leaf: u32) -> CpuidResult {
let mut ebx;
let ecx;
let edx;
- // SAFETY: Calling `__cpuid`` is safe on all x86_64 CPUs except for SGX,
+ // SAFETY: Calling `__cpuid` is safe on all x86_64 CPUs except for SGX,
// which doesn't support `cpuid`.
// https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/core_arch/src/x86/cpuid.rs#L102-L109
unsafe {
asm!(
- "mov {ebx_tmp:r}, rbx", // save rbx which is reserved by LLVM
+ "mov r8, rbx", // save rbx which is reserved by LLVM
"cpuid",
- "xchg {ebx_tmp:r}, rbx", // restore rbx
- ebx_tmp = out(reg) ebx,
+ "xchg r8, rbx", // restore rbx
+ out("r8") ebx,
inout("eax") leaf => eax,
inout("ecx") 0 => ecx,
out("edx") edx,
@@ -50,67 +50,31 @@ fn __cpuid(leaf: u32) -> CpuidResult {
CpuidResult { eax, ebx, ecx, edx }
}
-// https://en.wikipedia.org/wiki/CPUID
-const _VENDOR_ID_INTEL: [u32; 3] = _vender(b"GenuineIntel"); // Intel
-const _VENDOR_ID_INTEL2: [u32; 3] = _vender(b"GenuineIotel"); // Intel https://github.com/InstLatx64/InstLatx64/commit/8fdd319884c67d2c6ec1ca0c595b42c1c4b8d803
-const _VENDOR_ID_AMD: [u32; 3] = _vender(b"AuthenticAMD"); // AMD
-const _VENDOR_ID_CENTAUR: [u32; 3] = _vender(b"CentaurHauls"); // Centaur/VIA/Zhaoxin
-const _VENDOR_ID_ZHAOXIN: [u32; 3] = _vender(b" Shanghai "); // Zhaoxin
-const fn _vender(b: &[u8; 12]) -> [u32; 3] {
- [
- u32::from_ne_bytes([b[0], b[1], b[2], b[3]]),
- u32::from_ne_bytes([b[4], b[5], b[6], b[7]]),
- u32::from_ne_bytes([b[8], b[9], b[10], b[11]]),
- ]
-}
-fn _vendor_id() -> [u32; 3] {
- let CpuidResult { ebx, ecx, edx, .. } = __cpuid(0);
- [ebx, edx, ecx]
-}
-fn _vendor_has_vmovdqa_atomic(vendor_id: [u32; 3], family: u32) -> bool {
- // VMOVDQA is atomic on Intel, AMD, and Zhaoxin CPUs with AVX.
- // See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104688 for details.
- vendor_id == _VENDOR_ID_INTEL
- || vendor_id == _VENDOR_ID_INTEL2
- || vendor_id == _VENDOR_ID_AMD
- || vendor_id == _VENDOR_ID_ZHAOXIN
- || vendor_id == _VENDOR_ID_CENTAUR && family > 6
-}
-
#[cold]
fn _detect(info: &mut CpuInfo) {
- let CpuidResult {
- #[cfg(target_feature = "sse")]
- eax: proc_info_eax,
- ecx: proc_info_ecx,
- ..
- } = __cpuid(1);
+ let CpuidResult { ecx: proc_info_ecx, .. } = __cpuid(1);
- // https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/std_detect/src/detect/os/x86.rs#L111
+ // https://github.com/rust-lang/rust/blob/1.92.0/library/std_detect/src/detect/os/x86.rs#L104
if test(proc_info_ecx, 13) {
- info.set(CpuInfo::HAS_CMPXCHG16B);
+ info.set(CpuInfoFlag::cmpxchg16b);
}
- // We only use VMOVDQA when SSE is enabled. See atomic_load_vmovdqa() in atomic128/x86_64.rs for more.
+ // We only use VMOVDQA when SSE is enabled. See _atomic_load_vmovdqa() in atomic128/x86_64.rs for more.
#[cfg(target_feature = "sse")]
{
use core::arch::x86_64::_xgetbv;
- // https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/std_detect/src/detect/os/x86.rs#L131-L224
+ // https://github.com/rust-lang/rust/blob/1.92.0/library/std_detect/src/detect/os/x86.rs#L166-L236
let cpu_xsave = test(proc_info_ecx, 26);
if cpu_xsave {
let cpu_osxsave = test(proc_info_ecx, 27);
if cpu_osxsave {
- // SAFETY: Calling `_xgetbv`` is safe because the CPU has `xsave` support
+ // SAFETY: Calling `_xgetbv` is safe because the CPU has `xsave` support
// and OS has set `osxsave`.
let xcr0 = unsafe { _xgetbv(0) };
let os_avx_support = xcr0 & 6 == 6;
if os_avx_support && test(proc_info_ecx, 28) {
- let vendor_id = _vendor_id();
- let family = (proc_info_eax >> 8) & 0x0F;
- if _vendor_has_vmovdqa_atomic(vendor_id, family) {
- info.set(CpuInfo::HAS_VMOVDQA_ATOMIC);
- }
+ info.set(CpuInfoFlag::avx);
}
}
}
@@ -126,41 +90,20 @@ fn _detect(info: &mut CpuInfo) {
)]
#[cfg(test)]
mod tests {
- use std::{
- io::{self, Write},
- mem, str,
- };
-
use super::*;
#[test]
- #[cfg_attr(portable_atomic_test_outline_atomics_detect_false, ignore)]
+ #[cfg_attr(portable_atomic_test_detect_false, ignore = "detection disabled")]
fn test_cpuid() {
- assert_eq!(std::is_x86_feature_detected!("cmpxchg16b"), detect().has_cmpxchg16b());
- let vendor_id = _vendor_id();
- {
- let stdout = io::stderr();
- let mut stdout = stdout.lock();
- let _ = writeln!(
- stdout,
- "\n vendor_id: {} (ebx: {:x}, edx: {:x}, ecx: {:x})",
- str::from_utf8(&unsafe { mem::transmute::<[u32; 3], [u8; 12]>(vendor_id) })
- .unwrap(),
- vendor_id[0],
- vendor_id[1],
- vendor_id[2],
- );
- }
- let CpuidResult { eax: proc_info_eax, .. } = __cpuid(1);
- let family = (proc_info_eax >> 8) & 0x0F;
- if _vendor_has_vmovdqa_atomic(vendor_id, family) {
- assert_eq!(std::is_x86_feature_detected!("avx"), detect().has_vmovdqa_atomic());
- } else {
- assert!(!detect().has_vmovdqa_atomic());
- }
+ // The recent Rosetta 2 unofficially implements AVX support.
+ // (The OS reports it as unsupported, likely due to poor performance.)
+ #[cfg(target_vendor = "apple")]
assert_eq!(
- unsafe { mem::transmute::<[u32; 3], [u8; 12]>(_VENDOR_ID_INTEL) },
- *b"GenuineIntel"
+ std::is_x86_feature_detected!("avx"),
+ detect().avx() || cfg!(target_feature = "avx")
);
+ #[cfg(not(target_vendor = "apple"))]
+ assert_eq!(std::is_x86_feature_detected!("avx"), detect().avx());
+ assert_eq!(std::is_x86_feature_detected!("cmpxchg16b"), detect().cmpxchg16b());
}
}
### external/vendor/portable-atomic/src/imp/fallback/mod.rs
@@ -15,95 +15,56 @@ type and the value type must be the same.
#![cfg_attr(
any(
- all(
- target_arch = "x86_64",
- not(portable_atomic_no_outline_atomics),
- not(any(target_env = "sgx", miri)),
- ),
- all(
- target_arch = "powerpc64",
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- any(
- all(
- target_os = "linux",
- any(
- all(
- target_env = "gnu",
- any(target_endian = "little", not(target_feature = "crt-static")),
- ),
- all(
- any(target_env = "musl", target_env = "ohos", target_env = "uclibc"),
- not(target_feature = "crt-static"),
- ),
- portable_atomic_outline_atomics,
- ),
- ),
- target_os = "android",
- target_os = "freebsd",
- target_os = "openbsd",
- ),
- not(any(miri, portable_atomic_sanitize_thread)),
- ),
all(
target_arch = "riscv32",
not(any(miri, portable_atomic_sanitize_thread)),
- not(portable_atomic_no_asm),
- any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- any(test, portable_atomic_outline_atomics), // TODO(riscv): currently disabled by default
- any(target_os = "linux", target_os = "android"),
- ),
- ),
- ),
- all(
- target_arch = "riscv64",
- not(portable_atomic_no_asm),
- any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- all(
- feature = "fallback",
- not(portable_atomic_no_outline_atomics),
- any(test, portable_atomic_outline_atomics), // TODO(riscv): currently disabled by default
- any(target_os = "linux", target_os = "android"),
- not(any(miri, portable_atomic_sanitize_thread)),
- ),
- ),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ not(portable_atomic_no_outline_atomics),
+ any(target_os = "linux", target_os = "android"),
),
all(
target_arch = "arm",
+ not(any(miri, portable_atomic_sanitize_thread)),
any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
any(target_os = "linux", target_os = "android"),
+ any(test, not(any(target_feature = "v6", portable_atomic_target_feature = "v6"))),
not(portable_atomic_no_outline_atomics),
),
),
allow(dead_code)
)]
-#[macro_use]
-pub(crate) mod utils;
+// This module requires CAS and this crate only provides atomics up to 128-bit.
+// I don't believe there are any 16-bit multi-core systems with CAS, and
+// at least no such architecture is currently supported in Rust.
+// 128-bit targets that lack atomic usize CAS also do not reach this module.
+#[cfg(target_pointer_width = "16")]
+compile_error!(
+ "internal error: unreachable since atomics for 16-bit targets can always be provided by disable interrupts"
+);
+#[cfg(target_pointer_width = "128")]
+compile_error!(
+ "internal error: unreachable since 128-bit target either has atomic CAS for the pointer width or does not have CAS"
+);
+
+mod utils;
// Use "wide" sequence lock if the pointer width <= 32 for preventing its counter against wrap
// around.
//
-// In narrow architectures (pointer width <= 16), the counter is still <= 32-bit and may be
-// vulnerable to wrap around. But it's mostly okay, since in such a primitive hardware, the
-// counter will not be increased that fast.
-//
// Some 64-bit architectures have ABI with 32-bit pointer width (e.g., x86_64 X32 ABI,
// AArch64 ILP32 ABI, mips64 N32 ABI). On those targets, AtomicU64 is available and fast,
-// so use it to implement normal sequence lock.
+// so use it to implement normal sequence lock and reduce chunks of byte-wise atomic memcpy.
cfg_has_fast_atomic_64! {
mod seq_lock;
+ type AtomicChunk = core::sync::atomic::AtomicU64;
+ type Chunk = u64;
}
cfg_no_fast_atomic_64! {
#[path = "seq_lock_wide.rs"]
mod seq_lock;
+ type AtomicChunk = core::sync::atomic::AtomicU32;
+ type Chunk = u32;
}
use core::{cell::UnsafeCell, mem, sync::atomic::Ordering};
@@ -112,21 +73,16 @@ use self::{
seq_lock::{SeqLock, SeqLockWriteGuard},
utils::CachePadded,
};
+#[cfg(portable_atomic_no_strict_provenance)]
+use crate::utils::ptr::PtrExt as _;
+use crate::utils::unlikely;
-// Some 64-bit architectures have ABI with 32-bit pointer width (e.g., x86_64 X32 ABI,
-// AArch64 ILP32 ABI, mips64 N32 ABI). On those targets, AtomicU64 is fast,
-// so use it to reduce chunks of byte-wise atomic memcpy.
-use self::seq_lock::{AtomicChunk, Chunk};
-
-// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.7/crossbeam-utils/src/atomic/atomic_cell.rs#L969-L1016.
+// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/atomic/atomic_cell.rs#L970-L1010.
#[inline]
#[must_use]
fn lock(addr: usize) -> &'static SeqLock {
// The number of locks is a prime number because we want to make sure `addr % LEN` gets
// dispersed across all locks.
- //
- // crossbeam-utils 0.8.7 uses 97 here but does not use CachePadded,
- // so the actual concurrency level will be smaller.
const LEN: usize = 67;
const L: CachePadded<SeqLock> = CachePadded::new(SeqLock::new());
static LOCKS: [CachePadded<SeqLock>; LEN] = [
@@ -173,7 +129,7 @@ macro_rules! atomic {
// in SeqLock implementations:
//
// - https://github.com/Amanieu/seqlock/issues/2
- // - https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.7/crossbeam-utils/src/atomic/atomic_cell.rs#L1111-L1116
+ // - https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/atomic/atomic_cell.rs#L1063-L1069
// - https://rust-lang.zulipchat.com/#narrow/stream/136281-t-lang.2Fwg-unsafe-code-guidelines/topic/avoiding.20UB.20due.20to.20races.20by.20discarding.20result.3F
//
// However, in our use case, the implementation that loads/stores value as
@@ -216,37 +172,193 @@ macro_rules! atomic {
// SAFETY: any data races are prevented by the lock and atomic operation.
unsafe impl Sync for $atomic_type {}
- impl_default_no_fetch_ops!($atomic_type, $int_type);
- impl_default_bit_opts!($atomic_type, $int_type);
- impl $atomic_type {
- #[inline]
- pub(crate) const fn new(v: $int_type) -> Self {
- Self { v: UnsafeCell::new(v) }
- }
+ #[cfg(any(
+ test,
+ not(any(
+ all(
+ target_arch = "x86_64",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ portable_atomic_no_cmpxchg16b_intrinsic,
+ )),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ not(portable_atomic_no_outline_atomics),
+ not(any(target_env = "sgx", miri)),
+ ),
+ all(
+ target_arch = "powerpc64",
+ not(portable_atomic_no_asm),
+ not(portable_atomic_no_outline_atomics),
+ any(
+ all(
+ target_os = "linux",
+ any(
+ all(
+ target_env = "gnu",
+ any(target_endian = "little", not(target_feature = "crt-static")),
+ ),
+ all(
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
+ ),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
+ portable_atomic_outline_atomics,
+ ),
+ ),
+ target_os = "android",
+ all(
+ target_os = "freebsd",
+ any(
+ target_endian = "little",
+ not(target_feature = "crt-static"),
+ portable_atomic_outline_atomics,
+ ),
+ ),
+ target_os = "openbsd",
+ all(
+ target_os = "aix",
+ not(portable_atomic_pre_llvm_20),
+ portable_atomic_outline_atomics, // TODO(aix): currently disabled by default
+ ),
+ ),
+ not(any(miri, portable_atomic_sanitize_thread)),
+ ),
+ all(
+ target_arch = "riscv64",
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ not(portable_atomic_no_outline_atomics),
+ any(target_os = "linux", target_os = "android"),
+ ),
+ )),
+ ))]
+ items!({
+ impl_default_no_fetch_ops!($atomic_type, $int_type);
+ impl_default_bit_opts!($atomic_type, $int_type);
+ impl $atomic_type {
+ #[inline]
+ pub(crate) const fn new(v: $int_type) -> Self {
+ Self { v: UnsafeCell::new(v) }
+ }
- #[inline]
- pub(crate) fn is_lock_free() -> bool {
- Self::IS_ALWAYS_LOCK_FREE
- }
- pub(crate) const IS_ALWAYS_LOCK_FREE: bool = false;
+ #[inline]
+ pub(crate) fn is_lock_free() -> bool {
+ Self::IS_ALWAYS_LOCK_FREE
+ }
+ pub(crate) const IS_ALWAYS_LOCK_FREE: bool = false;
+
+ #[inline]
+ #[cfg_attr(
+ all(debug_assertions, not(portable_atomic_no_track_caller)),
+ track_caller
+ )]
+ pub(crate) fn compare_exchange_weak(
+ &self,
+ current: $int_type,
+ new: $int_type,
+ success: Ordering,
+ failure: Ordering,
+ ) -> Result<$int_type, $int_type> {
+ self.compare_exchange(current, new, success, failure)
+ }
+
+ #[inline]
+ pub(crate) fn not(&self, order: Ordering) {
+ self.fetch_not(order);
+ }
+ #[inline]
+ pub(crate) fn neg(&self, order: Ordering) {
+ self.fetch_neg(order);
+ }
+ #[inline]
+ pub(crate) const fn as_ptr(&self) -> *mut $int_type {
+ self.v.get()
+ }
+ }
+ });
+ #[cfg_attr(
+ any(
+ all(
+ target_arch = "x86_64",
+ not(all(
+ any(miri, portable_atomic_sanitize_thread),
+ portable_atomic_no_cmpxchg16b_intrinsic,
+ )),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ not(portable_atomic_no_outline_atomics),
+ not(any(target_env = "sgx", miri)),
+ ),
+ all(
+ target_arch = "powerpc64",
+ not(portable_atomic_no_asm),
+ not(portable_atomic_no_outline_atomics),
+ any(
+ all(
+ target_os = "linux",
+ any(
+ all(
+ target_env = "gnu",
+ any(target_endian = "little", not(target_feature = "crt-static")),
+ ),
+ all(
+ target_env = "musl",
+ any(not(target_feature = "crt-static"), feature = "std"),
+ ),
+ target_env = "ohos",
+ all(target_env = "uclibc", not(target_feature = "crt-static")),
+ portable_atomic_outline_atomics,
+ ),
+ ),
+ target_os = "android",
+ all(
+ target_os = "freebsd",
+ any(
+ target_endian = "little",
+ not(target_feature = "crt-static"),
+ portable_atomic_outline_atomics,
+ ),
+ ),
+ target_os = "openbsd",
+ all(
+ target_os = "aix",
+ not(portable_atomic_pre_llvm_20),
+ portable_atomic_outline_atomics, // TODO(aix): currently disabled by default
+ ),
+ ),
+ not(any(miri, portable_atomic_sanitize_thread)),
+ ),
+ all(
+ target_arch = "riscv64",
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ not(portable_atomic_no_outline_atomics),
+ any(target_os = "linux", target_os = "android"),
+ ),
+ ),
+ allow(dead_code)
+ )]
+ impl $atomic_type {
#[inline]
#[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
pub(crate) fn load(&self, order: Ordering) -> $int_type {
crate::utils::assert_load_ordering(order);
- let lock = lock(self.v.get() as usize);
+ let lock = lock(self.v.get().addr());
// Try doing an optimistic read first.
- if let Some(stamp) = lock.optimistic_read() {
+ if let Some(stamp) = lock.optimistic_read(order) {
let val = self.optimistic_read();
- if lock.validate_read(stamp) {
+ if lock.validate_read(stamp, order) {
return val;
}
}
// Grab a regular write lock so that writers don't starve this load.
- let guard = lock.write();
+ let guard = lock.write(
+ Ordering::AcqRel, // we already emit sc fence in optimistic_read if needed
+ );
let val = self.read(&guard);
// The value hasn't been changed. Drop the guard without incrementing the stamp.
guard.abort();
@@ -257,13 +369,13 @@ macro_rules! atomic {
#[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
pub(crate) fn store(&self, val: $int_type, order: Ordering) {
crate::utils::assert_store_ordering(order);
- let guard = lock(self.v.get() as usize).write();
+ let guard = lock(self.v.get().addr()).write(order);
self.write(val, &guard)
}
#[inline]
- pub(crate) fn swap(&self, val: $int_type, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn swap(&self, val: $int_type, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
self.write(val, &guard);
prev
@@ -279,7 +391,13 @@ macro_rules! atomic {
failure: Ordering,
) -> Result<$int_type, $int_type> {
crate::utils::assert_compare_exchange_ordering(success, failure);
- let guard = lock(self.v.get() as usize).write();
+ let order = if unlikely(success == Ordering::SeqCst || failure == Ordering::SeqCst)
+ {
+ Ordering::SeqCst
+ } else {
+ Ordering::AcqRel
+ };
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
if prev == current {
self.write(new, &guard);
@@ -292,114 +410,104 @@ macro_rules! atomic {
}
#[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn compare_exchange_weak(
- &self,
- current: $int_type,
- new: $int_type,
- success: Ordering,
- failure: Ordering,
- ) -> Result<$int_type, $int_type> {
- self.compare_exchange(current, new, success, failure)
- }
-
- #[inline]
- pub(crate) fn fetch_add(&self, val: $int_type, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_add(&self, val: $int_type, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
self.write(prev.wrapping_add(val), &guard);
prev
}
#[inline]
- pub(crate) fn fetch_sub(&self, val: $int_type, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_sub(&self, val: $int_type, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
self.write(prev.wrapping_sub(val), &guard);
prev
}
#[inline]
- pub(crate) fn fetch_and(&self, val: $int_type, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_and(&self, val: $int_type, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
self.write(prev & val, &guard);
prev
}
#[inline]
- pub(crate) fn fetch_nand(&self, val: $int_type, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_nand(&self, val: $int_type, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
self.write(!(prev & val), &guard);
prev
}
#[inline]
- pub(crate) fn fetch_or(&self, val: $int_type, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_or(&self, val: $int_type, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
self.write(prev | val, &guard);
prev
}
#[inline]
- pub(crate) fn fetch_xor(&self, val: $int_type, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_xor(&self, val: $int_type, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
self.write(prev ^ val, &guard);
prev
}
#[inline]
- pub(crate) fn fetch_max(&self, val: $int_type, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_not(&self, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
- self.write(core::cmp::max(prev, val), &guard);
+ self.write(!prev, &guard);
prev
}
#[inline]
- pub(crate) fn fetch_min(&self, val: $int_type, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_neg(&self, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
- self.write(core::cmp::min(prev, val), &guard);
+ self.write(prev.wrapping_neg(), &guard);
prev
}
-
+ }
+ impl $atomic_type {
#[inline]
- pub(crate) fn fetch_not(&self, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_max(&self, val: $int_type, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
- self.write(!prev, &guard);
+ self.write(core::cmp::max(prev, val), &guard);
prev
}
- #[inline]
- pub(crate) fn not(&self, order: Ordering) {
- self.fetch_not(order);
- }
#[inline]
- pub(crate) fn fetch_neg(&self, _order: Ordering) -> $int_type {
- let guard = lock(self.v.get() as usize).write();
+ pub(crate) fn fetch_min(&self, val: $int_type, order: Ordering) -> $int_type {
+ let guard = lock(self.v.get().addr()).write(order);
let prev = self.read(&guard);
- self.write(prev.wrapping_neg(), &guard);
+ self.write(core::cmp::min(prev, val), &guard);
prev
}
- #[inline]
- pub(crate) fn neg(&self, order: Ordering) {
- self.fetch_neg(order);
- }
-
- #[inline]
- pub(crate) const fn as_ptr(&self) -> *mut $int_type {
- self.v.get()
- }
}
};
}
-#[cfg_attr(portable_atomic_no_cfg_target_has_atomic, cfg(any(test, portable_atomic_no_atomic_64)))]
+#[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(
+ test,
+ not(any(
+ not(portable_atomic_no_atomic_64),
+ all(
+ target_arch = "riscv32",
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(target_feature = "zacas", portable_atomic_target_feature = "zacas"),
+ ),
+ ))
+ ))
+)]
#[cfg_attr(
not(portable_atomic_no_cfg_target_has_atomic),
cfg(any(
@@ -409,11 +517,8 @@ macro_rules! atomic {
all(
target_arch = "riscv32",
not(any(miri, portable_atomic_sanitize_thread)),
- not(portable_atomic_no_asm),
- any(
- target_feature = "experimental-zacas",
- portable_atomic_target_feature = "experimental-zacas",
- ),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(target_feature = "zacas", portable_atomic_target_feature = "zacas"),
),
))
))
### external/vendor/portable-atomic/src/imp/fallback/outline_atomics.rs
@@ -31,15 +31,15 @@ macro_rules! debug_assert_outline_atomics {
() => {
#[cfg(target_arch = "x86_64")]
{
- debug_assert!(!super::detect::detect().has_cmpxchg16b());
+ debug_assert!(!super::detect::detect().cmpxchg16b());
}
#[cfg(target_arch = "powerpc64")]
{
- debug_assert!(!super::detect::detect().has_quadword_atomics());
+ debug_assert!(!super::detect::detect().quadword_atomics());
}
#[cfg(any(target_arch = "riscv32", target_arch = "riscv64"))]
{
- debug_assert!(!super::detect::detect().has_zacas());
+ debug_assert!(!super::detect::detect().zacas());
}
#[cfg(target_arch = "arm")]
{
@@ -66,7 +66,7 @@ fn_alias! {
atomic_load_seqcst = atomic_load(Ordering::SeqCst);
}
-#[cfg(not(any(target_arch = "riscv32", target_arch = "riscv64")))]
+#[cfg(not(any(target_arch = "arm", target_arch = "riscv32", target_arch = "riscv64")))]
#[cold]
pub(crate) unsafe fn atomic_store(dst: *mut Udw, val: Udw, order: Ordering) {
debug_assert_outline_atomics!();
@@ -76,12 +76,11 @@ pub(crate) unsafe fn atomic_store(dst: *mut Udw, val: Udw, order: Ordering) {
(*(dst as *const AtomicUdw)).store(val, order);
}
}
-#[cfg(not(any(target_arch = "riscv32", target_arch = "riscv64")))]
+#[cfg(not(any(target_arch = "arm", target_arch = "riscv32", target_arch = "riscv64")))]
fn_alias! {
#[cold]
pub(crate) unsafe fn(dst: *mut Udw, val: Udw);
// fallback's atomic store has at least release semantics.
- #[cfg(not(target_arch = "arm"))]
atomic_store_non_seqcst = atomic_store(Ordering::Release);
atomic_store_seqcst = atomic_store(Ordering::SeqCst);
}
### external/vendor/portable-atomic/src/imp/fallback/seq_lock.rs
@@ -1,80 +1,92 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
-// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.7/crossbeam-utils/src/atomic/seq_lock.rs.
+// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/atomic/seq_lock_wide.rs.
use core::{
mem::ManuallyDrop,
- sync::atomic::{self, Ordering},
+ sync::atomic::{self, AtomicU64, Ordering},
};
-use super::utils::Backoff;
+use super::utils::{Backoff, sc_fence};
+#[cfg(portable_atomic_unsafe_assume_privileged)]
+use crate::imp::interrupt::arch as interrupt;
+use crate::utils::unlikely;
-// See mod.rs for details.
-#[cfg(any(target_pointer_width = "16", target_pointer_width = "32"))]
-pub(super) use core::sync::atomic::AtomicU64 as AtomicStamp;
-#[cfg(not(any(target_pointer_width = "16", target_pointer_width = "32")))]
-pub(super) use core::sync::atomic::AtomicUsize as AtomicStamp;
-#[cfg(not(any(target_pointer_width = "16", target_pointer_width = "32")))]
-pub(super) type Stamp = usize;
-#[cfg(any(target_pointer_width = "16", target_pointer_width = "32"))]
-pub(super) type Stamp = u64;
+pub(super) type State = u64;
-// See mod.rs for details.
-pub(super) type AtomicChunk = AtomicStamp;
-pub(super) type Chunk = Stamp;
+const LOCKED: State = 1;
/// A simple stamped lock.
pub(super) struct SeqLock {
/// The current state of the lock.
///
/// All bits except the least significant one hold the current stamp. When locked, the state
/// equals 1 and doesn't contain a valid stamp.
- state: AtomicStamp,
+ state: AtomicU64,
}
impl SeqLock {
#[inline]
pub(super) const fn new() -> Self {
- Self { state: AtomicStamp::new(0) }
+ Self { state: AtomicU64::new(0) }
}
/// If not locked, returns the current stamp.
///
/// This method should be called before optimistic reads.
#[inline]
- pub(super) fn optimistic_read(&self) -> Option<Stamp> {
- let state = self.state.load(Ordering::Acquire);
- if state == 1 {
- None
- } else {
- Some(state)
+ pub(super) fn optimistic_read(&self, order: Ordering) -> Option<State> {
+ if unlikely(order == Ordering::SeqCst) {
+ sc_fence();
}
+ let state = self.state.load(Ordering::Acquire);
+ if state == LOCKED { None } else { Some(state) }
}
/// Returns `true` if the current stamp is equal to `stamp`.
///
/// This method should be called after optimistic reads to check whether they are valid. The
/// argument `stamp` should correspond to the one returned by method `optimistic_read`.
#[inline]
- pub(super) fn validate_read(&self, stamp: Stamp) -> bool {
+ pub(super) fn validate_read(&self, stamp: State, order: Ordering) -> bool {
atomic::fence(Ordering::Acquire);
- self.state.load(Ordering::Relaxed) == stamp
+ let result = self.state.load(Ordering::Relaxed) == stamp;
+ if unlikely(order == Ordering::SeqCst) && result {
+ sc_fence();
+ }
+ result
}
/// Grabs the lock for writing.
#[inline]
- pub(super) fn write(&self) -> SeqLockWriteGuard<'_> {
+ pub(super) fn write(&self, order: Ordering) -> SeqLockWriteGuard<'_> {
+ let emit_sc_fence = order == Ordering::SeqCst;
+ if unlikely(emit_sc_fence) {
+ sc_fence();
+ }
+
+ // Get current interrupt state and disable interrupts when the user
+ // explicitly declares that privileged instructions are available.
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ let interrupt_state = interrupt::disable();
+
let mut backoff = Backoff::new();
loop {
- let previous = self.state.swap(1, Ordering::Acquire);
+ let previous = self.state.swap(LOCKED, Ordering::Acquire);
- if previous != 1 {
+ if previous != LOCKED {
atomic::fence(Ordering::Release);
- return SeqLockWriteGuard { lock: self, state: previous };
+ return SeqLockWriteGuard {
+ lock: self,
+ state: previous,
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ interrupt_state,
+ emit_sc_fence,
+ };
}
- while self.state.load(Ordering::Relaxed) == 1 {
+ while self.state.load(Ordering::Relaxed) == LOCKED {
backoff.snooze();
}
}
@@ -88,7 +100,13 @@ pub(super) struct SeqLockWriteGuard<'a> {
lock: &'a SeqLock,
/// The stamp before locking.
- state: Stamp,
+ state: State,
+
+ /// The interrupt state before disabling.
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ interrupt_state: interrupt::State,
+
+ emit_sc_fence: bool,
}
impl SeqLockWriteGuard<'_> {
@@ -103,6 +121,17 @@ impl SeqLockWriteGuard<'_> {
//
// Release ordering for synchronizing with `optimistic_read`.
this.lock.state.store(this.state, Ordering::Release);
+
+ // Restore interrupt state.
+ // SAFETY: the state was retrieved by the previous `disable`.
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ unsafe {
+ interrupt::restore(this.interrupt_state);
+ }
+
+ if unlikely(this.emit_sc_fence) {
+ sc_fence();
+ }
}
}
@@ -113,35 +142,50 @@ impl Drop for SeqLockWriteGuard<'_> {
//
// Release ordering for synchronizing with `optimistic_read`.
self.lock.state.store(self.state.wrapping_add(2), Ordering::Release);
+
+ // Restore interrupt state.
+ // SAFETY: the state was retrieved by the previous `disable`.
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ unsafe {
+ interrupt::restore(self.interrupt_state);
+ }
+
+ if unlikely(self.emit_sc_fence) {
+ sc_fence();
+ }
}
}
#[cfg(test)]
mod tests {
- use super::SeqLock;
+ use super::{Ordering, SeqLock};
#[test]
fn smoke() {
- let lock = SeqLock::new();
- let before = lock.optimistic_read().unwrap();
- assert!(lock.validate_read(before));
- {
- let _guard = lock.write();
+ for &order in &[Ordering::AcqRel, Ordering::SeqCst] {
+ let lock = SeqLock::new();
+ let before = lock.optimistic_read(order).unwrap();
+ assert!(lock.validate_read(before, order));
+ {
+ let _guard = lock.write(order);
+ }
+ assert!(!lock.validate_read(before, order));
+ let after = lock.optimistic_read(order).unwrap();
+ assert_ne!(before, after);
}
- assert!(!lock.validate_read(before));
- let after = lock.optimistic_read().unwrap();
- assert_ne!(before, after);
}
#[test]
fn test_abort() {
- let lock = SeqLock::new();
- let before = lock.optimistic_read().unwrap();
- {
- let guard = lock.write();
- guard.abort();
+ for &order in &[Ordering::AcqRel, Ordering::SeqCst] {
+ let lock = SeqLock::new();
+ let before = lock.optimistic_read(order).unwrap();
+ {
+ let guard = lock.write(order);
+ guard.abort();
+ }
+ let after = lock.optimistic_read(order).unwrap();
+ assert_eq!(before, after, "aborted write does not update the stamp");
}
- let after = lock.optimistic_read().unwrap();
- assert_eq!(before, after, "aborted write does not update the stamp");
}
}
### external/vendor/portable-atomic/src/imp/fallback/seq_lock_wide.rs
@@ -1,44 +1,50 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
-// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.7/crossbeam-utils/src/atomic/seq_lock_wide.rs.
+// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/atomic/seq_lock_wide.rs.
use core::{
mem::ManuallyDrop,
- sync::atomic::{self, AtomicUsize, Ordering},
+ sync::atomic::{self, AtomicU32, Ordering},
};
-use super::utils::Backoff;
+use super::utils::{Backoff, sc_fence};
+#[cfg(portable_atomic_unsafe_assume_privileged)]
+use crate::imp::interrupt::arch as interrupt;
+use crate::utils::unlikely;
-// See mod.rs for details.
-pub(super) type AtomicChunk = AtomicUsize;
-pub(super) type Chunk = usize;
+pub(super) type State = u32;
+
+const LOCKED: State = 1;
/// A simple stamped lock.
///
-/// The state is represented as two `AtomicUsize`: `state_hi` for high bits and `state_lo` for low
+/// The state is represented as two `AtomicU32`: `state_hi` for high bits and `state_lo` for low
/// bits.
pub(super) struct SeqLock {
/// The high bits of the current state of the lock.
- state_hi: AtomicUsize,
+ state_hi: AtomicU32,
/// The low bits of the current state of the lock.
///
/// All bits except the least significant one hold the current stamp. When locked, the state_lo
/// equals 1 and doesn't contain a valid stamp.
- state_lo: AtomicUsize,
+ state_lo: AtomicU32,
}
impl SeqLock {
#[inline]
pub(super) const fn new() -> Self {
- Self { state_hi: AtomicUsize::new(0), state_lo: AtomicUsize::new(0) }
+ Self { state_hi: AtomicU32::new(0), state_lo: AtomicU32::new(0) }
}
/// If not locked, returns the current stamp.
///
/// This method should be called before optimistic reads.
#[inline]
- pub(super) fn optimistic_read(&self) -> Option<(usize, usize)> {
+ pub(super) fn optimistic_read(&self, order: Ordering) -> Option<(State, State)> {
+ if unlikely(order == Ordering::SeqCst) {
+ sc_fence();
+ }
// The acquire loads from `state_hi` and `state_lo` synchronize with the release stores in
// `SeqLockWriteGuard::drop` and `SeqLockWriteGuard::abort`.
//
@@ -47,19 +53,15 @@ impl SeqLock {
// critical section of (`state_hi`, `state_lo`) happens before now.
let state_hi = self.state_hi.load(Ordering::Acquire);
let state_lo = self.state_lo.load(Ordering::Acquire);
- if state_lo == 1 {
- None
- } else {
- Some((state_hi, state_lo))
- }
+ if state_lo == LOCKED { None } else { Some((state_hi, state_lo)) }
}
/// Returns `true` if the current stamp is equal to `stamp`.
///
/// This method should be called after optimistic reads to check whether they are valid. The
/// argument `stamp` should correspond to the one returned by method `optimistic_read`.
#[inline]
- pub(super) fn validate_read(&self, stamp: (usize, usize)) -> bool {
+ pub(super) fn validate_read(&self, stamp: (State, State), order: Ordering) -> bool {
// Thanks to the fence, if we're noticing any modification to the data at the critical
// section of `(stamp.0, stamp.1)`, then the critical section's write of 1 to state_lo should be
// visible.
@@ -79,25 +81,45 @@ impl SeqLock {
// Except for the case that both `state_hi` and `state_lo` wrapped around, the following
// condition implies that we're noticing no modification to the data after the critical
// section of `(stamp.0, stamp.1)`.
- (state_hi, state_lo) == stamp
+ let result = (state_hi, state_lo) == stamp;
+ if unlikely(order == Ordering::SeqCst) && result {
+ sc_fence();
+ }
+ result
}
/// Grabs the lock for writing.
#[inline]
- pub(super) fn write(&self) -> SeqLockWriteGuard<'_> {
+ pub(super) fn write(&self, order: Ordering) -> SeqLockWriteGuard<'_> {
+ let emit_sc_fence = order == Ordering::SeqCst;
+ if unlikely(emit_sc_fence) {
+ sc_fence();
+ }
+
+ // Get current interrupt state and disable interrupts when the user
+ // explicitly declares that privileged instructions are available.
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ let interrupt_state = interrupt::disable();
+
let mut backoff = Backoff::new();
loop {
- let previous = self.state_lo.swap(1, Ordering::Acquire);
+ let previous = self.state_lo.swap(LOCKED, Ordering::Acquire);
- if previous != 1 {
+ if previous != LOCKED {
// To synchronize with the acquire fence in `validate_read` via any modification to
// the data at the critical section of `(state_hi, previous)`.
atomic::fence(Ordering::Release);
- return SeqLockWriteGuard { lock: self, state_lo: previous };
+ return SeqLockWriteGuard {
+ lock: self,
+ state_lo: previous,
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ interrupt_state,
+ emit_sc_fence,
+ };
}
- while self.state_lo.load(Ordering::Relaxed) == 1 {
+ while self.state_lo.load(Ordering::Relaxed) == LOCKED {
backoff.snooze();
}
}
@@ -111,7 +133,13 @@ pub(super) struct SeqLockWriteGuard<'a> {
lock: &'a SeqLock,
/// The stamp before locking.
- state_lo: usize,
+ state_lo: State,
+
+ /// The interrupt state before disabling.
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ interrupt_state: interrupt::State,
+
+ emit_sc_fence: bool,
}
impl SeqLockWriteGuard<'_> {
@@ -126,6 +154,17 @@ impl SeqLockWriteGuard<'_> {
//
// Release ordering for synchronizing with `optimistic_read`.
this.lock.state_lo.store(this.state_lo, Ordering::Release);
+
+ // Restore interrupt state.
+ // SAFETY: the state was retrieved by the previous `disable`.
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ unsafe {
+ interrupt::restore(this.interrupt_state);
+ }
+
+ if unlikely(this.emit_sc_fence) {
+ sc_fence();
+ }
}
}
@@ -146,35 +185,50 @@ impl Drop for SeqLockWriteGuard<'_> {
//
// Release ordering for synchronizing with `optimistic_read`.
self.lock.state_lo.store(state_lo, Ordering::Release);
+
+ // Restore interrupt state.
+ // SAFETY: the state was retrieved by the previous `disable`.
+ #[cfg(portable_atomic_unsafe_assume_privileged)]
+ unsafe {
+ interrupt::restore(self.interrupt_state);
+ }
+
+ if unlikely(self.emit_sc_fence) {
+ sc_fence();
+ }
}
}
#[cfg(test)]
mod tests {
- use super::SeqLock;
+ use super::{Ordering, SeqLock};
#[test]
fn smoke() {
- let lock = SeqLock::new();
- let before = lock.optimistic_read().unwrap();
- assert!(lock.validate_read(before));
- {
- let _guard = lock.write();
+ for &order in &[Ordering::AcqRel, Ordering::SeqCst] {
+ let lock = SeqLock::new();
+ let before = lock.optimistic_read(order).unwrap();
+ assert!(lock.validate_read(before, order));
+ {
+ let _guard = lock.write(order);
+ }
+ assert!(!lock.validate_read(before, order));
+ let after = lock.optimistic_read(order).unwrap();
+ assert_ne!(before, after);
}
- assert!(!lock.validate_read(before));
- let after = lock.optimistic_read().unwrap();
- assert_ne!(before, after);
}
#[test]
fn test_abort() {
- let lock = SeqLock::new();
- let before = lock.optimistic_read().unwrap();
- {
- let guard = lock.write();
- guard.abort();
+ for &order in &[Ordering::AcqRel, Ordering::SeqCst] {
+ let lock = SeqLock::new();
+ let before = lock.optimistic_read(order).unwrap();
+ {
+ let guard = lock.write(order);
+ guard.abort();
+ }
+ let after = lock.optimistic_read(order).unwrap();
+ assert_eq!(before, after, "aborted write does not update the stamp");
}
- let after = lock.optimistic_read().unwrap();
- assert_eq!(before, after, "aborted write does not update the stamp");
}
}
### external/vendor/portable-atomic/src/imp/fallback/utils.rs
@@ -2,7 +2,7 @@
use core::ops;
-// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/9384f1eb2b356364e201ad38545e03c837d55f3a/crossbeam-utils/src/cache_padded.rs.
+// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/cache_padded.rs.
/// Pads and aligns a value to the length of a cache line.
// Starting from Intel's Sandy Bridge, spatial prefetcher is now pulling pairs of 64-byte cache
// lines at a time, so we have to align to 128 bytes rather than 64.
@@ -11,7 +11,7 @@ use core::ops;
// - https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
// - https://github.com/facebook/folly/blob/1b5288e6eea6df074758f877c849b6e73bbb9fbb/folly/lang/Align.h#L107
//
-// ARM's big.LITTLE architecture has asymmetric cores and "big" cores have 128-byte cache line size.
+// aarch64/arm64ec's big.LITTLE architecture has asymmetric cores and "big" cores have 128-byte cache line size.
//
// Sources:
// - https://www.mono-project.com/news/2016/09/12/arm64-icache/
@@ -109,7 +109,7 @@ impl<T> ops::Deref for CachePadded<T> {
}
}
-// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.7/crossbeam-utils/src/backoff.rs.
+// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/backoff.rs.
// Adjusted to reduce spinning.
/// Performs exponential backoff in spin loops.
pub(crate) struct Backoff {
@@ -145,3 +145,21 @@ impl Backoff {
}
}
}
+
+#[inline]
+pub(crate) fn sc_fence() {
+ cfg_sel!({
+ #[cfg(all(
+ any(target_arch = "x86", target_arch = "x86_64"),
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ ))]
+ {
+ crate::imp::x86::sc_fence();
+ }
+ #[cfg(else)]
+ {
+ core::sync::atomic::fence(core::sync::atomic::Ordering::SeqCst);
+ }
+ });
+}
### external/vendor/portable-atomic/src/imp/float/aarch64.rs
@@ -0,0 +1,266 @@
+// SPDX-License-Identifier: Apache-2.0 OR MIT
+
+/*
+Atomic float implementation based on AArch64 with FEAT_LSFE.
+
+This module provides atomic float implementations using FEAT_LSFE instructions.
+
+Generated asm:
+- aarch64 (+lsfe) https://godbolt.org/z/7vaxeofv1
+*/
+
+#[cfg(not(portable_atomic_no_asm))]
+use core::arch::asm;
+use core::sync::atomic::Ordering;
+
+#[cfg(portable_atomic_unstable_f16)]
+use super::int::AtomicF16;
+#[cfg(portable_atomic_unstable_f128)]
+use super::int::AtomicF128;
+use super::int::{AtomicF32, AtomicF64};
+
+// TODO: optimize no return cases:
+// https://developer.arm.com/documentation/ddi0602/2025-06/SIMD-FP-Instructions/STFADD--STFADDL--Floating-point-atomic-add-in-memory--without-return-
+// https://developer.arm.com/documentation/ddi0602/2025-06/SIMD-FP-Instructions/STFMAXNM--STFMAXNML--Floating-point-atomic-maximum-number-in-memory--without-return-
+// https://developer.arm.com/documentation/ddi0602/2025-06/SIMD-FP-Instructions/STFMINNM--STFMINNML--Floating-point-atomic-minimum-number-in-memory--without-return-
+
+#[cfg(not(portable_atomic_pre_llvm_20))]
+macro_rules! start_lsfe {
+ () => {
+ ".arch_extension lsfe"
+ };
+}
+
+#[cfg(not(portable_atomic_pre_llvm_20))]
+macro_rules! atomic_rmw {
+ ($op:ident, $order:ident) => {
+ atomic_rmw!($op, $order, write = $order)
+ };
+ ($op:ident, $order:ident, write = $write:ident) => {
+ match $order {
+ Ordering::Relaxed => $op!("", "", ""),
+ Ordering::Acquire => $op!("a", "", ""),
+ Ordering::Release => $op!("", "l", ""),
+ Ordering::AcqRel => $op!("a", "l", ""),
+ // In MSVC environments, SeqCst stores/writes needs fences after writes.
+ // https://reviews.llvm.org/D141748
+ #[cfg(target_env = "msvc")]
+ Ordering::SeqCst if $write == Ordering::SeqCst => $op!("a", "l", "dmb ish"),
+ // AcqRel and SeqCst RMWs are equivalent in non-MSVC environments.
+ Ordering::SeqCst => $op!("a", "l", ""),
+ _ => unreachable!(),
+ }
+ };
+}
+#[cfg(portable_atomic_pre_llvm_20)]
+macro_rules! atomic_rmw_inst {
+ ($op:ident, $order:ident) => {
+ atomic_rmw_inst!($op, $order, write = $order)
+ };
+ ($op:ident, $order:ident, write = $write:ident) => {
+ match $order {
+ Ordering::Relaxed => $op!("2", ""), // ""
+ Ordering::Acquire => $op!("a", ""), // "a"
+ Ordering::Release => $op!("6", ""), // "l"
+ Ordering::AcqRel => $op!("e", ""), // "al"
+ // In MSVC environments, SeqCst stores/writes needs fences after writes.
+ // https://reviews.llvm.org/D141748
+ #[cfg(target_env = "msvc")]
+ Ordering::SeqCst if $write == Ordering::SeqCst => $op!("e", "dmb ish"),
+ // AcqRel and SeqCst RMWs are equivalent in non-MSVC environments.
+ Ordering::SeqCst => $op!("e", ""),
+ _ => unreachable!(),
+ }
+ };
+}
+
+macro_rules! atomic_float {
+ ($atomic_type:ident, $float_type:ident, $modifier:tt, $inst_modifier:tt) => {
+ impl $atomic_type {
+ #[inline]
+ pub(crate) fn fetch_add(&self, val: $float_type, order: Ordering) -> $float_type {
+ let dst = self.as_ptr();
+ let out;
+ // SAFETY: any data races are prevented by atomic intrinsics and the raw
+ // pointer passed in is valid because we got it from a reference.
+ //
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/SIMD-FP-Instructions/LDFADD--LDFADDA--LDFADDAL--LDFADDL--Floating-point-atomic-add-in-memory-
+ unsafe {
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ macro_rules! add {
+ ($acquire:tt, $release:tt, $fence:tt) => {
+ asm!(
+ start_lsfe!(),
+ concat!("ldfadd", $acquire, $release, " {out:", $modifier, "}, {val:", $modifier, "}, [{dst}]"),
+ $fence,
+ dst = in(reg) ptr_reg!(dst),
+ val = in(vreg) val,
+ out = lateout(vreg) out,
+ options(nostack),
+ )
+ };
+ }
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ atomic_rmw!(add, order);
+ // LLVM supports FEAT_LSFE instructions on LLVM 20+, so use .inst directive on old LLVM.
+ // https://github.com/llvm/llvm-project/commit/67ff5ba9af9754261abe11d762af11532a816126
+ #[cfg(portable_atomic_pre_llvm_20)]
+ macro_rules! add {
+ ($order:tt, $fence:tt) => {
+ asm!(
+ // ldfadd{,a,l,al} {h,s,d}0, {h,s,d}1, [x2]
+ concat!(".inst 0x", $inst_modifier, "c", $order, "00041"),
+ $fence,
+ in("x2") ptr_reg!(dst),
+ in("v1") val,
+ out("v0") out,
+ options(nostack),
+ )
+ };
+ }
+ #[cfg(portable_atomic_pre_llvm_20)]
+ atomic_rmw_inst!(add, order);
+ }
+ out
+ }
+ #[inline]
+ pub(crate) fn fetch_sub(&self, val: $float_type, order: Ordering) -> $float_type {
+ // There is no atomic sub instruction, so add `-val`.
+ self.fetch_add(-val, order)
+ }
+ #[inline]
+ pub(crate) fn fetch_max(&self, val: $float_type, order: Ordering) -> $float_type {
+ let dst = self.as_ptr();
+ let out;
+ // SAFETY: any data races are prevented by atomic intrinsics and the raw
+ // pointer passed in is valid because we got it from a reference.
+ //
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/SIMD-FP-Instructions/LDFMAXNM--LDFMAXNMA--LDFMAXNMAL--LDFMAXNML--Floating-point-atomic-maximum-number-in-memory-
+ unsafe {
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ macro_rules! max {
+ ($acquire:tt, $release:tt, $fence:tt) => {
+ asm!(
+ start_lsfe!(),
+ concat!("ldfmaxnm", $acquire, $release, " {out:", $modifier, "}, {val:", $modifier, "}, [{dst}]"),
+ $fence,
+ dst = in(reg) ptr_reg!(dst),
+ val = in(vreg) val,
+ out = lateout(vreg) out,
+ options(nostack),
+ )
+ };
+ }
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ atomic_rmw!(max, order);
+ // LLVM supports FEAT_LSFE instructions on LLVM 20+, so use .inst directive on old LLVM.
+ // https://github.com/llvm/llvm-project/commit/67ff5ba9af9754261abe11d762af11532a816126
+ #[cfg(portable_atomic_pre_llvm_20)]
+ macro_rules! max {
+ ($order:tt, $fence:tt) => {
+ asm!(
+ // ldfmaxnm{,a,l,al} {h,s,d}0, {h,s,d}1, [x2]
+ concat!(".inst 0x", $inst_modifier, "c", $order, "06041"),
+ $fence,
+ in("x2") ptr_reg!(dst),
+ in("v1") val,
+ out("v0") out,
+ options(nostack),
+ )
+ };
+ }
+ #[cfg(portable_atomic_pre_llvm_20)]
+ atomic_rmw_inst!(max, order);
+ }
+ out
+ }
+ #[inline]
+ pub(crate) fn fetch_min(&self, val: $float_type, order: Ordering) -> $float_type {
+ let dst = self.as_ptr();
+ let out;
+ // SAFETY: any data races are prevented by atomic intrinsics and the raw
+ // pointer passed in is valid because we got it from a reference.
+ //
+ // Refs: https://developer.arm.com/documentation/ddi0602/2025-06/SIMD-FP-Instructions/LDFMINNM--LDFMINNMA--LDFMINNMAL--LDFMINNML--Floating-point-atomic-minimum-number-in-memory-
+ unsafe {
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ macro_rules! min {
+ ($acquire:tt, $release:tt, $fence:tt) => {
+ asm!(
+ start_lsfe!(),
+ concat!("ldfminnm", $acquire, $release, " {out:", $modifier, "}, {val:", $modifier, "}, [{dst}]"),
+ $fence,
+ dst = in(reg) ptr_reg!(dst),
+ val = in(vreg) val,
+ out = lateout(vreg) out,
+ options(nostack),
+ )
+ };
+ }
+ #[cfg(not(portable_atomic_pre_llvm_20))]
+ atomic_rmw!(min, order);
+ // LLVM supports FEAT_LSFE instructions on LLVM 20+, so use .inst directive on old LLVM.
+ // https://github.com/llvm/llvm-project/commit/67ff5ba9af9754261abe11d762af11532a816126
+ #[cfg(portable_atomic_pre_llvm_20)]
+ macro_rules! min {
+ ($order:tt, $fence:tt) => {
+ asm!(
+ // ldfminnm{,a,l,al} {h,s,d}0, {h,s,d}1, [x2]
+ concat!(".inst 0x", $inst_modifier, "c", $order, "07041"),
+ $fence,
+ in("x2") ptr_reg!(dst),
+ in("v1") val,
+ out("v0") out,
+ options(nostack),
+ )
+ };
+ }
+ #[cfg(portable_atomic_pre_llvm_20)]
+ atomic_rmw_inst!(min, order);
+ }
+ out
+ }
+ }
+ };
+}
+
+#[cfg(portable_atomic_unstable_f16)]
+atomic_float!(AtomicF16, f16, "h", "7");
+atomic_float!(AtomicF32, f32, "s", "b");
+atomic_float!(AtomicF64, f64, "d", "f");
+
+#[cfg(portable_atomic_unstable_f128)]
+impl AtomicF128 {
+ #[inline]
+ pub(crate) fn fetch_add(&self, val: f128, order: Ordering) -> f128 {
+ self.fetch_update_(order, |x| x + val)
+ }
+ #[inline]
+ pub(crate) fn fetch_sub(&self, val: f128, order: Ordering) -> f128 {
+ self.fetch_update_(order, |x| x - val)
+ }
+ #[inline]
+ pub(super) fn fetch_update_<F>(&self, order: Ordering, mut f: F) -> f128
+ where
+ F: FnMut(f128) -> f128,
+ {
+ // This is a private function and all instances of `f` only operate on the value
+ // loaded, so there is no need to synchronize the first load/failed CAS.
+ let mut prev = self.load(Ordering::Relaxed);
+ loop {
+ let next = f(prev);
+ match self.compare_exchange_weak(prev, next, order, Ordering::Relaxed) {
+ Ok(x) => return x,
+ Err(next_prev) => prev = next_prev,
+ }
+ }
+ }
+ #[inline]
+ pub(crate) fn fetch_max(&self, val: f128, order: Ordering) -> f128 {
+ self.fetch_update_(order, |x| x.max(val))
+ }
+ #[inline]
+ pub(crate) fn fetch_min(&self, val: f128, order: Ordering) -> f128 {
+ self.fetch_update_(order, |x| x.min(val))
+ }
+}
### external/vendor/portable-atomic/src/imp/float/int.rs
@@ -1,15 +1,16 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
/*
-AtomicF{32,64} implementation based on AtomicU{32,64}.
+Atomic float implementation based on atomic integer.
This module provides atomic float implementations using atomic integer.
Note that most of `fetch_*` operations of atomic floats are implemented using
CAS loops, which can be slower than equivalent operations of atomic integers.
-GPU targets have atomic instructions for float, so GPU targets will use
-architecture-specific implementations instead of this implementation in the
+AArch64 with FEAT_LSFE and GPU targets have atomic instructions for float.
+See aarch64.rs for AArch64 with FEAT_LSFE.
+GPU targets will also use architecture-specific implementations instead of this implementation in the
future: https://github.com/taiki-e/portable-atomic/issues/34 / https://github.com/taiki-e/portable-atomic/pull/45
*/
@@ -90,8 +91,22 @@ macro_rules! atomic_float {
pub(crate) fn swap(&self, val: $float_type, order: Ordering) -> $float_type {
$float_type::from_bits(self.as_bits().swap(val.to_bits(), order))
}
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_neg(&self, order: Ordering) -> $float_type {
+ const NEG_MASK: $int_type = !0 / 2 + 1;
+ $float_type::from_bits(self.as_bits().fetch_xor(NEG_MASK, order))
+ }
- cfg_has_atomic_cas! {
+ #[inline]
+ #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
+ pub(crate) fn fetch_abs(&self, order: Ordering) -> $float_type {
+ const ABS_MASK: $int_type = !0 / 2;
+ $float_type::from_bits(self.as_bits().fetch_and(ABS_MASK, order))
+ }
+ }
+ cfg_has_atomic_cas! {
+ impl $atomic_type {
#[inline]
#[cfg_attr(
any(all(debug_assertions, not(portable_atomic_no_track_caller)), miri),
@@ -137,7 +152,15 @@ macro_rules! atomic_float {
Err(v) => Err($float_type::from_bits(v)),
}
}
-
+ }
+ #[cfg(not(all(
+ any(target_arch = "aarch64", target_arch = "arm64ec"),
+ any(target_feature = "lsfe", portable_atomic_target_feature = "lsfe"),
+ target_feature = "neon", // for vreg
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ )))]
+ impl $atomic_type {
#[inline]
#[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
pub(crate) fn fetch_add(&self, val: $float_type, order: Ordering) -> $float_type {
@@ -179,28 +202,23 @@ macro_rules! atomic_float {
pub(crate) fn fetch_min(&self, val: $float_type, order: Ordering) -> $float_type {
self.fetch_update_(order, |x| x.min(val))
}
- } // cfg_has_atomic_cas!
- #[inline]
- #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
- pub(crate) fn fetch_neg(&self, order: Ordering) -> $float_type {
- const NEG_MASK: $int_type = !0 / 2 + 1;
- $float_type::from_bits(self.as_bits().fetch_xor(NEG_MASK, order))
- }
-
- #[inline]
- #[cfg_attr(miri, track_caller)] // even without panics, this helps for Miri backtraces
- pub(crate) fn fetch_abs(&self, order: Ordering) -> $float_type {
- const ABS_MASK: $int_type = !0 / 2;
- $float_type::from_bits(self.as_bits().fetch_and(ABS_MASK, order))
- }
}
+ } // cfg_has_atomic_cas!
} // cfg_has_atomic_cas_or_amo32!
};
}
+#[cfg(portable_atomic_unstable_f16)]
+cfg_has_atomic_16! {
+ atomic_float!(AtomicF16, f16, AtomicU16, u16, 2);
+}
cfg_has_atomic_32! {
atomic_float!(AtomicF32, f32, AtomicU32, u32, 4);
}
cfg_has_atomic_64! {
atomic_float!(AtomicF64, f64, AtomicU64, u64, 8);
}
+#[cfg(portable_atomic_unstable_f128)]
+cfg_has_atomic_128! {
+ atomic_float!(AtomicF128, f128, AtomicU128, u128, 16);
+}
### external/vendor/portable-atomic/src/imp/float/mod.rs
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: Apache-2.0 OR MIT
+
+/*
+Atomic float implementations
+*/
+
+#![allow(clippy::float_arithmetic)]
+
+mod int;
+
+#[cfg(all(
+ any(target_arch = "aarch64", target_arch = "arm64ec"),
+ any(target_feature = "lsfe", portable_atomic_target_feature = "lsfe"),
+ target_feature = "neon", // for vreg
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+))]
+mod aarch64;
+
+#[cfg(portable_atomic_unstable_f16)]
+cfg_has_atomic_16! {
+ pub(crate) use self::int::AtomicF16;
+}
+cfg_has_atomic_32! {
+ pub(crate) use self::int::AtomicF32;
+}
+cfg_has_atomic_64! {
+ pub(crate) use self::int::AtomicF64;
+}
+#[cfg(portable_atomic_unstable_f128)]
+cfg_has_atomic_128! {
+ pub(crate) use self::int::AtomicF128;
+}
### external/vendor/portable-atomic/src/imp/interrupt/README.md
@@ -1,9 +1,14 @@
-# Implementation of disabling interrupts
+# Fallback implementation based on disabling interrupts or critical-section
-This module is used to provide atomic CAS for targets where atomic CAS is not available in the standard library.
+This module supports two different critical section implementations:
-- On MSP430 and AVR, they are always single-core and has no unprivileged mode, so this module is always used.
-- On Armv6-M (thumbv6m), pre-v6 Arm (e.g., thumbv4t, thumbv5te), RISC-V without A-extension, and Xtensa, they could be multi-core, so this module is used when the `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg) is enabled.
+- Built-in "disable all interrupts".
+ - On MSP430 and AVR, they are always single-core and has no unprivileged mode, so this is enabled by default.
+ - On Armv6-M (thumbv6m), pre-v6 Arm (e.g., thumbv4t, thumbv5te), RISC-V without A-extension, and Xtensa, they could be multi-core or unprivileged mode, so this is enabled when the user explicitly declares that the system is single-core and that privileged instructions are available using `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg).
+- Call into the `critical-section` crate (which allows the user to plug any implementation).
+ - This is enabled when the user asks for it with the [`critical-section` feature](../../../README.md#optional-features-critical-section).
+
+`unsafe-assume-privileged` feature (`portable_atomic_unsafe_assume_privileged` cfg) also uses this module's interrupt disable implementation part.
The `unsafe-assume-single-core` implementation uses privileged instructions to disable interrupts, so it usually doesn't work on unprivileged mode.
Enabling this feature in an environment where privileged instructions are not available, or if the instructions used are not sufficient to disable interrupts in the system, it is also usually considered **unsound**, although the details are system-dependent.
@@ -12,20 +17,21 @@ Consider using the [`critical-section` feature](../../../README.md#optional-feat
For some targets, the implementation can be changed by explicitly enabling features.
-- On Armv6-M, this disables interrupts by modifying the PRIMASK register.
-- On pre-v6 Arm, this disables interrupts by modifying the I (IRQ mask) bit of the CPSR.
-- On pre-v6 Arm with the `disable-fiq` feature (or `portable_atomic_disable_fiq` cfg), this disables interrupts by modifying the I (IRQ mask) bit and F (FIQ mask) bit of the CPSR.
-- On RISC-V (without A-extension), this disables interrupts by modifying the MIE (Machine Interrupt Enable) bit of the `mstatus` register.
-- On RISC-V (without A-extension) with the `s-mode` feature (or `portable_atomic_s_mode` cfg), this disables interrupts by modifying the SIE (Supervisor Interrupt Enable) bit of the `sstatus` register.
-- On RISC-V (without A-extension) with the `zaamo` target feature (or `force-amo` feature or `portable_atomic_force_amo` cfg), this uses AMO instructions for RMWs that have corresponding AMO instructions even if A-extension is disabled. For other RMWs, this disables interrupts as usual.
+- On Arm M-Profile architectures, this disables interrupts by modifying the PRIMASK register.
+- On Arm (except for M-Profile architectures), this disables interrupts by modifying the I (IRQ mask) bit of the CPSR.
+- On Arm (except for M-Profile architectures) with the `disable-fiq` feature (or `portable_atomic_disable_fiq` cfg), this disables interrupts by modifying the I (IRQ mask) bit and F (FIQ mask) bit of the CPSR.
+- On RISC-V, this disables interrupts by modifying the MIE (Machine Interrupt Enable) bit of the `mstatus` register.
+- On RISC-V with the `s-mode` feature (or `portable_atomic_s_mode` cfg), this disables interrupts by modifying the SIE (Supervisor Interrupt Enable) bit of the `sstatus` register.
+- On RISC-V with the `zaamo` target feature (or `force-amo` feature or `portable_atomic_force_amo` cfg), this uses AMO instructions for RMWs that have corresponding AMO instructions even if A-extension is disabled. For other RMWs, this disables interrupts as usual.
- On MSP430, this disables interrupts by modifying the GIE (Global Interrupt Enable) bit of the status register (SR).
- On AVR, this disables interrupts by modifying the I (Global Interrupt Enable) bit of the status register (SREG).
- On Xtensa, this disables interrupts by modifying the PS special register.
-Some operations don't require disabling interrupts:
+<a name="no-disable-interrupts"></a>Some operations don't require disabling interrupts:
- On architectures except for AVR: loads and stores with pointer size or smaller
- On AVR: 8-bit loads and stores
+- On AVR with `rmw` target feature additionally: 8-bit `swap`
- On MSP430 additionally: {8,16}-bit `add,sub,and,or,xor,not`
- On RISC-V with the `zaamo` target feature (or `portable_atomic_target_feature="zaamo"` cfg or `force-amo` feature or `portable_atomic_force_amo` cfg) additionally: 32-bit(RV32)/{32,64}-bit(RV64) `swap,fetch_{add,sub,and,or,xor,not,max,min},add,sub,and,or,xor,not`, {8,16}-bit `fetch_{and,or,xor,not},and,or,xor,not`[^1], and all operations of `AtomicBool`
### external/vendor/portable-atomic/src/imp/interrupt/armv4t.rs
@@ -1,10 +1,11 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
/*
+Arm A-Profile Architectures, Arm R-Profile Architectures, Legacy Arm Architectures
+
Refs: https://developer.arm.com/documentation/ddi0406/cb/System-Level-Architecture/The-System-Level-Programmers--Model/ARM-processor-modes-and-ARM-core-registers/Program-Status-Registers--PSRs-
-Generated asm:
-- armv5te https://godbolt.org/z/fhaW3d9Kv
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
#[cfg(not(portable_atomic_no_asm))]
@@ -26,20 +27,23 @@ macro_rules! mask {
};
}
-pub(super) type State = u32;
+pub(crate) type State = u32;
/// Disables interrupts and returns the previous interrupt state.
#[inline]
-#[instruction_set(arm::a32)]
-pub(super) fn disable() -> State {
+#[cfg_attr(
+ not(any(target_feature = "v7", portable_atomic_target_feature = "v7")),
+ instruction_set(arm::a32)
+)]
+pub(crate) fn disable() -> State {
let cpsr: State;
// SAFETY: reading CPSR and disabling interrupts are safe.
// (see module-level comments of interrupt/mod.rs on the safety of using privileged instructions)
unsafe {
asm!(
- "mrs {prev}, cpsr",
- concat!("orr {new}, {prev}, ", mask!()),
- "msr cpsr_c, {new}",
+ "mrs {prev}, cpsr", // prev = CPSR
+ concat!("orr {new}, {prev}, ", mask!()), // new = prev | mask
+ "msr cpsr_c, {new}", // CPSR.{I,F,T,M} = new.{I,F,T,M}
prev = out(reg) cpsr,
new = out(reg) _,
// Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
@@ -55,33 +59,42 @@ pub(super) fn disable() -> State {
///
/// The state must be the one retrieved by the previous `disable`.
#[inline]
-#[instruction_set(arm::a32)]
-pub(super) unsafe fn restore(cpsr: State) {
+#[cfg_attr(
+ not(any(target_feature = "v7", portable_atomic_target_feature = "v7")),
+ instruction_set(arm::a32)
+)]
+pub(crate) unsafe fn restore(prev_cpsr: State) {
// SAFETY: the caller must guarantee that the state was retrieved by the previous `disable`,
//
- // This clobbers the control field mask byte of CPSR. See msp430.rs to safety on this.
- // (preserves_flags is fine because we only clobber the I, F, T, and M bits of CPSR.)
+ // This clobbers the control field mask byte of CPSR. See msp430.rs for safety on this.
+ // (preserves_flags is fine because we can clobber only the I, F, T, and M bits of CPSR.)
//
// Refs: https://developer.arm.com/documentation/dui0473/m/arm-and-thumb-instructions/msr--general-purpose-register-to-psr-
unsafe {
- // Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
- asm!("msr cpsr_c, {0}", in(reg) cpsr, options(nostack, preserves_flags));
+ asm!(
+ "msr cpsr_c, {prev_cpsr}", // CPSR.{I,F,T,M} = prev_cpsr.{I,F,T,M}
+ prev_cpsr = in(reg) prev_cpsr,
+ // Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
+ options(nostack, preserves_flags),
+ );
}
}
// On pre-v6 Arm, we cannot use core::sync::atomic here because they call the
// `__sync_*` builtins for non-relaxed load/store (because pre-v6 Arm doesn't
// have Data Memory Barrier).
-//
-// Generated asm:
-// - armv5te https://godbolt.org/z/deqTqPzqz
-pub(crate) mod atomic {
+#[cfg_attr(portable_atomic_no_cfg_target_has_atomic, cfg(any(test, portable_atomic_no_atomic_cas)))]
+#[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(test, not(target_has_atomic = "ptr")))
+)]
+pub(super) mod atomic {
#[cfg(not(portable_atomic_no_asm))]
use core::arch::asm;
use core::{cell::UnsafeCell, sync::atomic::Ordering};
macro_rules! atomic {
- ($([$($generics:tt)*])? $atomic_type:ident, $value_type:ty, $asm_suffix:tt) => {
+ ($([$($generics:tt)*])? $atomic_type:ident, $value_type:ty, $suffix:tt) => {
#[repr(transparent)]
pub(crate) struct $atomic_type $(<$($generics)*>)? {
v: UnsafeCell<$value_type>,
@@ -105,7 +118,7 @@ pub(crate) mod atomic {
// And compiler fence is fine because the user explicitly declares that
// the system is single-core by using an unsafe cfg.
asm!(
- concat!("ldr", $asm_suffix, " {out}, [{src}]"),
+ concat!("ldr", $suffix, " {out}, [{src}]"), // atomic { out = *src }
src = in(reg) src,
out = lateout(reg) out,
options(nostack, preserves_flags),
@@ -124,7 +137,7 @@ pub(crate) mod atomic {
// And compiler fence is fine because the user explicitly declares that
// the system is single-core by using an unsafe cfg.
asm!(
- concat!("str", $asm_suffix, " {val}, [{dst}]"),
+ concat!("str", $suffix, " {val}, [{dst}]"), // atomic { *dst = val }
dst = in(reg) dst,
val = in(reg) val,
options(nostack, preserves_flags),
### external/vendor/portable-atomic/src/imp/interrupt/armv6m.rs
@@ -1,31 +1,41 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
/*
+Arm M-Profile Architectures
+
Adapted from https://github.com/rust-embedded/cortex-m.
-Generated asm:
-- armv6-m https://godbolt.org/z/1sqKnsY6n
+Refs: https://developer.arm.com/documentation/ddi0419/c/System-Level-Architecture/System-Level-Programmers--Model/Registers/The-special-purpose-mask-register--PRIMASK
+
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
#[cfg(not(portable_atomic_no_asm))]
use core::arch::asm;
-
+#[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(test, portable_atomic_no_atomic_cas))
+)]
+#[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(test, not(target_has_atomic = "ptr")))
+)]
pub(super) use core::sync::atomic;
-pub(super) type State = u32;
+pub(crate) type State = u32;
/// Disables interrupts and returns the previous interrupt state.
#[inline(always)]
-pub(super) fn disable() -> State {
+pub(crate) fn disable() -> State {
let primask: State;
// SAFETY: reading the priority mask register and disabling interrupts are safe.
// (see module-level comments of interrupt/mod.rs on the safety of using privileged instructions)
unsafe {
- // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
asm!(
- "mrs {0}, PRIMASK",
- "cpsid i",
- out(reg) primask,
+ "mrs {primask}, PRIMASK", // primask = PRIMASK
+ "cpsid i", // PRIMASK.PM = 1
+ primask = out(reg) primask,
+ // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
options(nostack, preserves_flags),
);
}
@@ -38,13 +48,17 @@ pub(super) fn disable() -> State {
///
/// The state must be the one retrieved by the previous `disable`.
#[inline(always)]
-pub(super) unsafe fn restore(primask: State) {
- if primask & 0x1 == 0 {
- // SAFETY: the caller must guarantee that the state was retrieved by the previous `disable`,
- // and we've checked that interrupts were enabled before disabling interrupts.
- unsafe {
+pub(crate) unsafe fn restore(prev_primask: State) {
+ // SAFETY: the caller must guarantee that the state was retrieved by the previous `disable`,
+ // and we've checked that interrupts were enabled before disabling interrupts.
+ //
+ // This clobbers the entire PRIMASK register. See msp430.rs for safety on this.
+ unsafe {
+ asm!(
+ "msr PRIMASK, {prev_primask}", // PRIMASK = prev_primask
+ prev_primask = in(reg) prev_primask,
// Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
- asm!("cpsie i", options(nostack, preserves_flags));
- }
+ options(nostack, preserves_flags),
+ );
}
}
### external/vendor/portable-atomic/src/imp/interrupt/avr.rs
@@ -9,33 +9,40 @@ Refs:
- AVR® Instruction Set Manual, Rev. DS40002198B
https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-InstructionSet-Manual-DS40002198.pdf
-Generated asm:
-- avr https://godbolt.org/z/W5jxGsToc
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
#[cfg(not(portable_atomic_no_asm))]
use core::arch::asm;
#[cfg(not(portable_atomic_no_asm))]
+#[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(test, portable_atomic_no_atomic_cas))
+)]
+#[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(test, not(target_has_atomic = "ptr")))
+)]
pub(super) use super::super::avr as atomic;
-pub(super) type State = u8;
+pub(crate) type State = u8;
/// Disables interrupts and returns the previous interrupt state.
#[inline(always)]
-pub(super) fn disable() -> State {
+pub(crate) fn disable() -> State {
let sreg: State;
// SAFETY: reading the status register (SREG) and disabling interrupts are safe.
// (see module-level comments of interrupt/mod.rs on the safety of using privileged instructions)
unsafe {
- // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
- // Do not use `preserves_flags` because CLI modifies the I bit of the status register (SREG).
// Refs: https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-InstructionSet-Manual-DS40002198.pdf#page=58
#[cfg(not(portable_atomic_no_asm))]
asm!(
"in {sreg}, 0x3F", // sreg = SREG
"cli", // SREG.I = 0
sreg = out(reg) sreg,
+ // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
+ // Do not use `preserves_flags` because CLI modifies the I bit of the status register (SREG).
options(nostack),
);
#[cfg(portable_atomic_no_asm)]
@@ -53,17 +60,17 @@ pub(super) fn disable() -> State {
///
/// The state must be the one retrieved by the previous `disable`.
#[inline(always)]
-pub(super) unsafe fn restore(prev_sreg: State) {
- // SAFETY: the caller must guarantee that the state was retrieved by the previous `disable`,
+pub(crate) unsafe fn restore(prev_sreg: State) {
+ // SAFETY: the caller must guarantee that the state was retrieved by the previous `disable`.
+ //
+ // This clobbers the entire status register. See msp430.rs for safety on this.
unsafe {
- // This clobbers the entire status register. See msp430.rs to safety on this.
- //
- // Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
- // Do not use `preserves_flags` because OUT modifies the status register (SREG).
#[cfg(not(portable_atomic_no_asm))]
asm!(
"out 0x3F, {prev_sreg}", // SREG = prev_sreg
prev_sreg = in(reg) prev_sreg,
+ // Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
+ // Do not use `preserves_flags` because OUT modifies the status register (SREG).
options(nostack),
);
#[cfg(portable_atomic_no_asm)]
### external/vendor/portable-atomic/src/imp/interrupt/mod.rs
@@ -1,52 +1,14 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
/*
-Critical section based fallback implementations
+Fallback implementation based on disabling interrupts or critical-section
-This module supports two different critical section implementations:
-- Built-in "disable all interrupts".
-- Call into the `critical-section` crate (which allows the user to plug any implementation).
+- mod.rs contains critical section based fallback implementations.
+- Each architecture modules contain implementations of disabling interrupts.
-The `critical-section`-based fallback is enabled when the user asks for it with the `critical-section`
-Cargo feature.
-
-The "disable interrupts" fallback is not sound on multi-core systems.
-Also, this uses privileged instructions to disable interrupts, so it usually
-doesn't work on unprivileged mode. Using this fallback in an environment where privileged
-instructions are not available is also usually considered **unsound**,
-although the details are system-dependent.
-
-Therefore, this implementation will only be enabled in one of the following cases:
-
-- When the user explicitly declares that the system is single-core and that
- privileged instructions are available using an unsafe cfg.
-- When we can safely assume that the system is single-core and that
- privileged instructions are available on the system.
-
-AVR, which is single core[^avr1] and LLVM also generates code that disables
-interrupts [^avr2] in atomic ops by default, is considered the latter.
-MSP430 as well.
-
-See also README.md of this directory.
-
-[^avr1]: https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/AVR/AVRExpandPseudoInsts.cpp#L1074
-[^avr2]: https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/test/CodeGen/AVR/atomics/load16.ll#L5
+See README.md of this directory for details.
*/
-// On some platforms, atomic load/store can be implemented in a more efficient
-// way than disabling interrupts. On MSP430, some RMWs that do not return the
-// previous value can also be optimized.
-//
-// Note: On single-core systems, it is okay to use critical session-based
-// CAS together with atomic load/store. The load/store will not be
-// called while interrupts are disabled, and since the load/store is
-// atomic, it is not affected by interrupts even if interrupts are enabled.
-#[cfg(not(any(
- all(target_arch = "avr", portable_atomic_no_asm),
- feature = "critical-section",
-)))]
-use self::arch::atomic;
-
#[cfg(not(feature = "critical-section"))]
#[cfg_attr(
all(
@@ -66,863 +28,658 @@ use self::arch::atomic;
#[cfg_attr(target_arch = "msp430", path = "msp430.rs")]
#[cfg_attr(any(target_arch = "riscv32", target_arch = "riscv64"), path = "riscv.rs")]
#[cfg_attr(target_arch = "xtensa", path = "xtensa.rs")]
-mod arch;
-
-use core::{cell::UnsafeCell, sync::atomic::Ordering};
-
-// Critical section implementations might use locks internally.
-#[cfg(feature = "critical-section")]
-const IS_ALWAYS_LOCK_FREE: bool = false;
-// Consider atomic operations based on disabling interrupts on single-core
-// systems are lock-free. (We consider the pre-v6 Arm Linux's atomic operations
-// provided in a similar way by the Linux kernel to be lock-free.)
-#[cfg(not(feature = "critical-section"))]
-const IS_ALWAYS_LOCK_FREE: bool = true;
-
-#[cfg(feature = "critical-section")]
-#[inline]
-fn with<F, R>(f: F) -> R
-where
- F: FnOnce() -> R,
-{
- critical_section::with(|_| f())
-}
-#[cfg(not(feature = "critical-section"))]
-#[inline(always)]
-fn with<F, R>(f: F) -> R
-where
- F: FnOnce() -> R,
-{
- // Get current interrupt state and disable interrupts
- let state = arch::disable();
-
- let r = f();
-
- // Restore interrupt state
- // SAFETY: the state was retrieved by the previous `disable`.
- unsafe { arch::restore(state) }
-
- r
-}
-
-#[cfg_attr(target_pointer_width = "16", repr(C, align(2)))]
-#[cfg_attr(target_pointer_width = "32", repr(C, align(4)))]
-#[cfg_attr(target_pointer_width = "64", repr(C, align(8)))]
-#[cfg_attr(target_pointer_width = "128", repr(C, align(16)))]
-pub(crate) struct AtomicPtr<T> {
- p: UnsafeCell<*mut T>,
-}
+pub(super) mod arch;
-// SAFETY: any data races are prevented by disabling interrupts or
-// atomic intrinsics (see module-level comments).
-unsafe impl<T> Send for AtomicPtr<T> {}
-// SAFETY: any data races are prevented by disabling interrupts or
-// atomic intrinsics (see module-level comments).
-unsafe impl<T> Sync for AtomicPtr<T> {}
-
-impl<T> AtomicPtr<T> {
- #[inline]
- pub(crate) const fn new(p: *mut T) -> Self {
- Self { p: UnsafeCell::new(p) }
- }
-
- #[inline]
- pub(crate) fn is_lock_free() -> bool {
- Self::IS_ALWAYS_LOCK_FREE
- }
- pub(crate) const IS_ALWAYS_LOCK_FREE: bool = IS_ALWAYS_LOCK_FREE;
-
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn load(&self, order: Ordering) -> *mut T {
- crate::utils::assert_load_ordering(order);
- #[cfg(not(any(target_arch = "avr", feature = "critical-section")))]
- {
- self.as_native().load(order)
- }
- #[cfg(any(target_arch = "avr", feature = "critical-section"))]
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe { self.p.get().read() })
- }
-
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn store(&self, ptr: *mut T, order: Ordering) {
- crate::utils::assert_store_ordering(order);
- #[cfg(not(any(target_arch = "avr", feature = "critical-section")))]
- {
- self.as_native().store(ptr, order);
- }
- #[cfg(any(target_arch = "avr", feature = "critical-section"))]
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe { self.p.get().write(ptr) });
- }
-
- #[inline]
- pub(crate) fn swap(&self, ptr: *mut T, order: Ordering) -> *mut T {
- let _ = order;
- #[cfg(all(
- any(target_arch = "riscv32", target_arch = "riscv64"),
- not(feature = "critical-section"),
- any(
- portable_atomic_force_amo,
- target_feature = "zaamo",
- portable_atomic_target_feature = "zaamo",
- ),
- ))]
- {
- self.as_native().swap(ptr, order)
- }
- #[cfg(not(all(
- any(target_arch = "riscv32", target_arch = "riscv64"),
- not(feature = "critical-section"),
- any(
- portable_atomic_force_amo,
- target_feature = "zaamo",
- portable_atomic_target_feature = "zaamo",
- ),
- )))]
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.p.get().read();
- self.p.get().write(ptr);
- prev
- })
- }
-
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn compare_exchange(
- &self,
- current: *mut T,
- new: *mut T,
- success: Ordering,
- failure: Ordering,
- ) -> Result<*mut T, *mut T> {
- crate::utils::assert_compare_exchange_ordering(success, failure);
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.p.get().read();
- if prev == current {
- self.p.get().write(new);
- Ok(prev)
- } else {
- Err(prev)
+#[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(test, portable_atomic_no_atomic_cas, portable_atomic_unsafe_assume_single_core))
+)]
+#[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(test, not(target_has_atomic = "ptr"), portable_atomic_unsafe_assume_single_core))
+)]
+items!({
+ use core::{cell::UnsafeCell, sync::atomic::Ordering};
+
+ // critical-section implementations might use locks internally.
+ #[cfg(feature = "critical-section")]
+ const IS_ALWAYS_LOCK_FREE: bool = false;
+ // Consider atomic operations based on disabling interrupts on single-core
+ // systems are lock-free. (We consider the pre-v6 Arm Linux's atomic operations
+ // provided in a similar way by the Linux kernel to be lock-free.)
+ #[cfg(not(feature = "critical-section"))]
+ const IS_ALWAYS_LOCK_FREE: bool = true;
+
+ // Put this in its own module to prevent guard creation.
+ use self::guard::disable;
+ mod guard {
+ // Note: The caller must NOT explicitly modify registers containing fields modified by disable/restore.
+ // (Fields modified as side effects of other operations are covered by the absence of preserves_flags,
+ // so they are fine -- see msp430.rs for more.)
+ #[inline(always)]
+ pub(super) fn disable() -> Guard {
+ Guard {
+ #[cfg(feature = "critical-section")]
+ // SAFETY: the state will be restored in the subsequent `release`.
+ state: unsafe { critical_section::acquire() },
+ #[cfg(not(feature = "critical-section"))]
+ // Get current interrupt state and disable interrupts.
+ state: super::arch::disable(),
}
- })
- }
-
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn compare_exchange_weak(
- &self,
- current: *mut T,
- new: *mut T,
- success: Ordering,
- failure: Ordering,
- ) -> Result<*mut T, *mut T> {
- self.compare_exchange(current, new, success, failure)
- }
-
- #[inline]
- pub(crate) const fn as_ptr(&self) -> *mut *mut T {
- self.p.get()
- }
-
- #[cfg(not(any(target_arch = "avr", feature = "critical-section")))]
- #[inline(always)]
- fn as_native(&self) -> &atomic::AtomicPtr<T> {
- // SAFETY: AtomicPtr and atomic::AtomicPtr have the same layout and
- // guarantee atomicity in a compatible way. (see module-level comments)
- unsafe { &*(self as *const Self as *const atomic::AtomicPtr<T>) }
- }
-}
-
-macro_rules! atomic_int {
- (base, $atomic_type:ident, $int_type:ident, $align:literal) => {
- #[repr(C, align($align))]
- pub(crate) struct $atomic_type {
- v: UnsafeCell<$int_type>,
}
-
- // Send is implicitly implemented.
- // SAFETY: any data races are prevented by disabling interrupts or
- // atomic intrinsics (see module-level comments).
- unsafe impl Sync for $atomic_type {}
-
- impl $atomic_type {
- #[inline]
- pub(crate) const fn new(v: $int_type) -> Self {
- Self { v: UnsafeCell::new(v) }
- }
-
- #[inline]
- pub(crate) fn is_lock_free() -> bool {
- Self::IS_ALWAYS_LOCK_FREE
- }
- pub(crate) const IS_ALWAYS_LOCK_FREE: bool = IS_ALWAYS_LOCK_FREE;
-
- #[inline]
- pub(crate) const fn as_ptr(&self) -> *mut $int_type {
- self.v.get()
- }
+ pub(super) struct Guard {
+ #[cfg(feature = "critical-section")]
+ state: critical_section::RestoreState,
+ #[cfg(not(feature = "critical-section"))]
+ state: super::arch::State,
}
- };
- (load_store_atomic $([$kind:ident])?, $atomic_type:ident, $int_type:ident, $align:literal) => {
- atomic_int!(base, $atomic_type, $int_type, $align);
- #[cfg(all(
- any(target_arch = "riscv32", target_arch = "riscv64"),
- not(feature = "critical-section"),
- any(
- portable_atomic_force_amo,
- target_feature = "zaamo",
- portable_atomic_target_feature = "zaamo",
- ),
- ))]
- atomic_int!(cas $([$kind])?, $atomic_type, $int_type);
- #[cfg(not(all(
- any(target_arch = "riscv32", target_arch = "riscv64"),
- not(feature = "critical-section"),
- any(
- portable_atomic_force_amo,
- target_feature = "zaamo",
- portable_atomic_target_feature = "zaamo",
- ),
- )))]
- atomic_int!(cas[emulate], $atomic_type, $int_type);
- impl $atomic_type {
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn load(&self, order: Ordering) -> $int_type {
- crate::utils::assert_load_ordering(order);
- #[cfg(not(any(
- all(target_arch = "avr", portable_atomic_no_asm),
- feature = "critical-section",
- )))]
- {
- self.as_native().load(order)
- }
- #[cfg(any(
- all(target_arch = "avr", portable_atomic_no_asm),
- feature = "critical-section",
- ))]
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe { self.v.get().read() })
- }
-
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn store(&self, val: $int_type, order: Ordering) {
- crate::utils::assert_store_ordering(order);
- #[cfg(not(any(
- all(target_arch = "avr", portable_atomic_no_asm),
- feature = "critical-section",
- )))]
- {
- self.as_native().store(val, order);
- }
- #[cfg(any(
- all(target_arch = "avr", portable_atomic_no_asm),
- feature = "critical-section",
- ))]
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe { self.v.get().write(val) });
- }
-
- #[cfg(not(any(
- all(target_arch = "avr", portable_atomic_no_asm),
- feature = "critical-section",
- )))]
+ impl Drop for Guard {
#[inline(always)]
- fn as_native(&self) -> &atomic::$atomic_type {
- // SAFETY: $atomic_type and atomic::$atomic_type have the same layout and
- // guarantee atomicity in a compatible way. (see module-level comments)
- unsafe { &*(self as *const Self as *const atomic::$atomic_type) }
- }
- }
-
- #[cfg(not(all(target_arch = "msp430", not(feature = "critical-section"))))]
- impl_default_no_fetch_ops!($atomic_type, $int_type);
- impl_default_bit_opts!($atomic_type, $int_type);
- #[cfg(not(all(target_arch = "msp430", not(feature = "critical-section"))))]
- impl $atomic_type {
- #[inline]
- pub(crate) fn not(&self, order: Ordering) {
- self.fetch_not(order);
- }
- }
- #[cfg(all(target_arch = "msp430", not(feature = "critical-section")))]
- impl $atomic_type {
- #[inline]
- pub(crate) fn add(&self, val: $int_type, order: Ordering) {
- self.as_native().add(val, order);
- }
- #[inline]
- pub(crate) fn sub(&self, val: $int_type, order: Ordering) {
- self.as_native().sub(val, order);
- }
- #[inline]
- pub(crate) fn and(&self, val: $int_type, order: Ordering) {
- self.as_native().and(val, order);
- }
- #[inline]
- pub(crate) fn or(&self, val: $int_type, order: Ordering) {
- self.as_native().or(val, order);
- }
- #[inline]
- pub(crate) fn xor(&self, val: $int_type, order: Ordering) {
- self.as_native().xor(val, order);
- }
- #[inline]
- pub(crate) fn not(&self, order: Ordering) {
- self.as_native().not(order);
+ fn drop(&mut self) {
+ #[cfg(feature = "critical-section")]
+ // SAFETY: the state was retrieved by the previous `acquire`.
+ unsafe {
+ critical_section::release(self.state);
+ }
+ #[cfg(not(feature = "critical-section"))]
+ // Restore interrupt state.
+ // SAFETY: the state was retrieved by the previous `disable`.
+ unsafe {
+ super::arch::restore(self.state);
+ }
}
}
- };
- (all_critical_session, $atomic_type:ident, $int_type:ident, $align:literal) => {
- atomic_int!(base, $atomic_type, $int_type, $align);
- atomic_int!(cas[emulate], $atomic_type, $int_type);
- impl_default_no_fetch_ops!($atomic_type, $int_type);
- impl_default_bit_opts!($atomic_type, $int_type);
- impl $atomic_type {
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn load(&self, order: Ordering) -> $int_type {
- crate::utils::assert_load_ordering(order);
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe { self.v.get().read() })
- }
+ }
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn store(&self, val: $int_type, order: Ordering) {
- crate::utils::assert_store_ordering(order);
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe { self.v.get().write(val) });
- }
+ macro_rules! atomic_base {
+ (base, $([$($generics:tt)*])? $atomic_type:ident, $value_type:ty, $align:literal) => {
+ #[repr(C, align($align))]
+ pub(crate) struct $atomic_type $(<$($generics)*>)? {
+ v: UnsafeCell<$value_type>,
+ }
+
+ // Send is implicitly implemented for atomic integers, but not for atomic pointers.
+ // SAFETY: any data races are prevented by disabling interrupts (or
+ // atomic intrinsics) or critical-section (see module-level comments).
+ unsafe impl $(<$($generics)*>)? Send for $atomic_type $(<$($generics)*>)? {}
+ // SAFETY: any data races are prevented by disabling interrupts (or
+ // atomic intrinsics) or critical-section (see module-level comments).
+ unsafe impl $(<$($generics)*>)? Sync for $atomic_type $(<$($generics)*>)? {}
+
+ impl $(<$($generics)*>)? $atomic_type $(<$($generics)*>)? {
+ #[inline]
+ pub(crate) const fn new(v: $value_type) -> Self {
+ Self { v: UnsafeCell::new(v) }
+ }
- #[inline]
- pub(crate) fn not(&self, order: Ordering) {
- self.fetch_not(order);
- }
- }
- };
- (cas[emulate], $atomic_type:ident, $int_type:ident) => {
- impl $atomic_type {
- #[inline]
- pub(crate) fn swap(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(val);
- prev
- })
- }
+ #[inline]
+ pub(crate) fn is_lock_free() -> bool {
+ Self::IS_ALWAYS_LOCK_FREE
+ }
+ pub(crate) const IS_ALWAYS_LOCK_FREE: bool = IS_ALWAYS_LOCK_FREE;
+
+ #[inline]
+ fn read(&self, _guard: &guard::Guard) -> $value_type {
+ // SAFETY: any data races are prevented by disabling interrupts or critical-section (see
+ // module-level comments) and the raw pointer is valid because we got it
+ // from a reference.
+ unsafe { self.v.get().read() }
+ }
+ #[inline]
+ fn write(&self, val: $value_type, _guard: &guard::Guard) {
+ // SAFETY: any data races are prevented by disabling interrupts or critical-section (see
+ // module-level comments) and the raw pointer is valid because we got it
+ // from a reference.
+ unsafe { self.v.get().write(val) }
+ }
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn compare_exchange(
- &self,
- current: $int_type,
- new: $int_type,
- success: Ordering,
- failure: Ordering,
- ) -> Result<$int_type, $int_type> {
- crate::utils::assert_compare_exchange_ordering(success, failure);
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
+ // As for CAS, there is no corresponding atomic operation on all architectures that use this code.
+ // (If the CAS instruction exists, all atomic operations can be implemented by it, so this code will not be used.)
+ #[inline]
+ #[cfg_attr(
+ all(debug_assertions, not(portable_atomic_no_track_caller)),
+ track_caller
+ )]
+ pub(crate) fn compare_exchange(
+ &self,
+ current: $value_type,
+ new: $value_type,
+ success: Ordering,
+ failure: Ordering,
+ ) -> Result<$value_type, $value_type> {
+ crate::utils::assert_compare_exchange_ordering(success, failure);
+ let guard = disable();
+ let prev = self.read(&guard);
if prev == current {
- self.v.get().write(new);
+ self.write(new, &guard);
Ok(prev)
} else {
Err(prev)
}
- })
- }
+ }
+ #[inline]
+ #[cfg_attr(
+ all(debug_assertions, not(portable_atomic_no_track_caller)),
+ track_caller
+ )]
+ pub(crate) fn compare_exchange_weak(
+ &self,
+ current: $value_type,
+ new: $value_type,
+ success: Ordering,
+ failure: Ordering,
+ ) -> Result<$value_type, $value_type> {
+ self.compare_exchange(current, new, success, failure)
+ }
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn compare_exchange_weak(
- &self,
- current: $int_type,
- new: $int_type,
- success: Ordering,
- failure: Ordering,
- ) -> Result<$int_type, $int_type> {
- self.compare_exchange(current, new, success, failure)
+ #[inline]
+ pub(crate) const fn as_ptr(&self) -> *mut $value_type {
+ self.v.get()
+ }
}
-
- #[inline]
- pub(crate) fn fetch_add(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev.wrapping_add(val));
- prev
- })
+ };
+ (native_load_store, $([$($generics:tt)*])? $atomic_type:ident, $value_type:ty) => {
+ impl $(<$($generics)*>)? core::ops::Deref for $atomic_type $(<$($generics)*>)? {
+ type Target = atomic::$atomic_type $(<$($generics)*>)?;
+ #[inline(always)]
+ fn deref(&self) -> &Self::Target {
+ // SAFETY: $atomic_type and atomic::$atomic_type have the same layout and
+ // guarantee atomicity in a compatible way. (see module-level comments)
+ unsafe {
+ &*(self as *const Self as *const atomic::$atomic_type $(<$($generics)*>)?)
+ }
+ }
}
-
- #[inline]
- pub(crate) fn fetch_sub(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev.wrapping_sub(val));
- prev
- })
+ };
+ (emulate_load_store, $([$($generics:tt)*])? $atomic_type:ident, $value_type:ty) => {
+ impl $(<$($generics)*>)? $atomic_type $(<$($generics)*>)? {
+ #[inline]
+ #[cfg_attr(
+ all(debug_assertions, not(portable_atomic_no_track_caller)),
+ track_caller
+ )]
+ pub(crate) fn load(&self, order: Ordering) -> $value_type {
+ crate::utils::assert_load_ordering(order);
+ let guard = disable();
+ self.read(&guard)
+ }
+ #[inline]
+ #[cfg_attr(
+ all(debug_assertions, not(portable_atomic_no_track_caller)),
+ track_caller
+ )]
+ pub(crate) fn store(&self, val: $value_type, order: Ordering) {
+ crate::utils::assert_store_ordering(order);
+ let guard = disable();
+ self.write(val, &guard);
+ }
}
-
- #[inline]
- pub(crate) fn fetch_and(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev & val);
+ };
+ (emulate_swap, $([$($generics:tt)*])? $atomic_type:ident, $value_type:ty) => {
+ impl $(<$($generics)*>)? $atomic_type $(<$($generics)*>)? {
+ #[inline]
+ pub(crate) fn swap(&self, val: $value_type, _order: Ordering) -> $value_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(val, &guard);
prev
- })
+ }
}
+ };
+ }
- #[inline]
- pub(crate) fn fetch_nand(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(!(prev & val));
- prev
- })
- }
+ #[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(
+ test,
+ target_arch = "avr",
+ target_arch = "msp430",
+ portable_atomic_no_atomic_cas
+ ))
+ )]
+ #[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(
+ test,
+ target_arch = "avr",
+ target_arch = "msp430",
+ not(target_has_atomic = "ptr")
+ ))
+ )]
+ items!({
+ #[cfg(target_pointer_width = "16")]
+ atomic_base!(base, [T] AtomicPtr, *mut T, 2);
+ #[cfg(target_pointer_width = "32")]
+ atomic_base!(base, [T] AtomicPtr, *mut T, 4);
+ #[cfg(target_pointer_width = "64")]
+ atomic_base!(base, [T] AtomicPtr, *mut T, 8);
+ #[cfg(target_pointer_width = "128")]
+ atomic_base!(base, [T] AtomicPtr, *mut T, 16);
+
+ impl_default_bit_opts!(AtomicPtr, usize);
+
+ cfg_sel!({
+ #[cfg(any(target_arch = "avr", feature = "critical-section"))]
+ {
+ atomic_base!(emulate_load_store, [T] AtomicPtr, *mut T);
+ }
+ #[cfg(else)]
+ {
+ atomic_base!(native_load_store, [T] AtomicPtr, *mut T);
+ }
+ });
- #[inline]
- pub(crate) fn fetch_or(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev | val);
+ #[cfg(not(all(
+ any(target_arch = "riscv32", target_arch = "riscv64"),
+ not(feature = "critical-section"),
+ any(
+ portable_atomic_force_amo,
+ target_feature = "zaamo",
+ portable_atomic_target_feature = "zaamo",
+ ),
+ )))]
+ items!({
+ atomic_base!(emulate_swap, [T] AtomicPtr, *mut T);
+ impl<T> AtomicPtr<T> {
+ #[inline]
+ pub(crate) fn fetch_byte_add(&self, val: usize, _order: Ordering) -> *mut T {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev.with_addr(prev.addr().wrapping_add(val)), &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_xor(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev ^ val);
+ }
+ #[inline]
+ pub(crate) fn fetch_byte_sub(&self, val: usize, _order: Ordering) -> *mut T {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev.with_addr(prev.addr().wrapping_sub(val)), &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_max(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(core::cmp::max(prev, val));
+ }
+ #[inline]
+ pub(crate) fn fetch_and(&self, val: usize, _order: Ordering) -> *mut T {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev.with_addr(prev.addr() & val), &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_min(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(core::cmp::min(prev, val));
+ }
+ #[inline]
+ pub(crate) fn fetch_or(&self, val: usize, _order: Ordering) -> *mut T {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev.with_addr(prev.addr() | val), &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_not(&self, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(!prev);
+ }
+ #[inline]
+ pub(crate) fn fetch_xor(&self, val: usize, _order: Ordering) -> *mut T {
+ #[cfg(portable_atomic_no_strict_provenance)]
+ use crate::utils::ptr::PtrExt as _;
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev.with_addr(prev.addr() ^ val), &guard);
prev
- })
+ }
}
+ });
+ impl<T> AtomicPtr<T> {
+ #[cfg(test)]
#[inline]
- pub(crate) fn fetch_neg(&self, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev.wrapping_neg());
- prev
- })
+ fn fetch_ptr_add(&self, val: usize, order: Ordering) -> *mut T {
+ self.fetch_byte_add(val.wrapping_mul(core::mem::size_of::<T>()), order)
}
+ #[cfg(test)]
#[inline]
- pub(crate) fn neg(&self, order: Ordering) {
- self.fetch_neg(order);
+ fn fetch_ptr_sub(&self, val: usize, order: Ordering) -> *mut T {
+ self.fetch_byte_sub(val.wrapping_mul(core::mem::size_of::<T>()), order)
}
}
- };
- // RISC-V 32-bit(RV32)/{32,64}-bit(RV64) RMW with Zaamo extension
- // RISC-V 8-bit/16-bit RMW with Zabha extension
- (cas, $atomic_type:ident, $int_type:ident) => {
- impl $atomic_type {
- #[inline]
- pub(crate) fn swap(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().swap(val, order)
- }
-
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn compare_exchange(
- &self,
- current: $int_type,
- new: $int_type,
- success: Ordering,
- failure: Ordering,
- ) -> Result<$int_type, $int_type> {
- crate::utils::assert_compare_exchange_ordering(success, failure);
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- if prev == current {
- self.v.get().write(new);
- Ok(prev)
- } else {
- Err(prev)
- }
- })
- }
-
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn compare_exchange_weak(
- &self,
- current: $int_type,
- new: $int_type,
- success: Ordering,
- failure: Ordering,
- ) -> Result<$int_type, $int_type> {
- self.compare_exchange(current, new, success, failure)
- }
-
- #[inline]
- pub(crate) fn fetch_add(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_add(val, order)
- }
- #[inline]
- pub(crate) fn fetch_sub(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_sub(val, order)
- }
- #[inline]
- pub(crate) fn fetch_and(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_and(val, order)
- }
-
- #[inline]
- pub(crate) fn fetch_nand(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(!(prev & val));
+ });
+
+ macro_rules! atomic_int {
+ (base, $atomic_type:ident, $int_type:ty, $align:literal) => {
+ atomic_base!(base, $atomic_type, $int_type, $align);
+ // As for nand and neg, there is no corresponding atomic operation on all architectures that use this code.
+ impl $atomic_type {
+ #[inline]
+ pub(crate) fn fetch_nand(&self, val: $int_type, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(!(prev & val), &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_or(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_or(val, order)
- }
- #[inline]
- pub(crate) fn fetch_xor(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_xor(val, order)
- }
- #[inline]
- pub(crate) fn fetch_max(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_max(val, order)
- }
- #[inline]
- pub(crate) fn fetch_min(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_min(val, order)
- }
- #[inline]
- pub(crate) fn fetch_not(&self, order: Ordering) -> $int_type {
- self.as_native().fetch_not(order)
- }
-
- #[inline]
- pub(crate) fn fetch_neg(&self, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev.wrapping_neg());
+ }
+ #[inline]
+ pub(crate) fn fetch_neg(&self, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev.wrapping_neg(), &guard);
prev
- })
+ }
+ #[inline]
+ pub(crate) fn neg(&self, order: Ordering) {
+ self.fetch_neg(order);
+ }
}
- #[inline]
- pub(crate) fn neg(&self, order: Ordering) {
- self.fetch_neg(order);
+ };
+ (load_store_atomic $([$kind:ident])?, $atomic_type:ident, $int_type:ty, $align:literal) => {
+ cfg_sel!({
+ #[cfg(feature = "critical-section")]
+ {
+ atomic_int!(all_critical_session, $atomic_type, $int_type, $align);
+ }
+ #[cfg(else)]
+ {
+ atomic_int!(base, $atomic_type, $int_type, $align);
+ impl_default_bit_opts!($atomic_type, $int_type);
+ // load/store
+ cfg_sel!({
+ // AVR with very old rustc
+ #[cfg(all(target_arch = "avr", portable_atomic_no_asm))]
+ {
+ atomic_base!(emulate_load_store, $atomic_type, $int_type);
+ }
+ #[cfg(else)]
+ {
+ atomic_base!(native_load_store, $atomic_type, $int_type);
+ }
+ });
+ // RMW
+ cfg_sel!({
+ // AVR 8-bit RMW with RMW instructions
+ #[cfg(all(
+ target_arch = "avr",
+ not(portable_atomic_no_asm),
+ any(target_feature = "rmw", portable_atomic_target_feature = "rmw"),
+ ))]
+ {
+ atomic_int!(emulate_arithmetic, $atomic_type, $int_type);
+ atomic_int!(emulate_bit, $atomic_type, $int_type);
+ }
+ // RISC-V RMW with Zaamo extension
+ #[cfg(all(
+ any(target_arch = "riscv32", target_arch = "riscv64"),
+ any(
+ portable_atomic_force_amo,
+ target_feature = "zaamo",
+ portable_atomic_target_feature = "zaamo",
+ ),
+ ))]
+ {
+ atomic_int!(cas $([$kind])?, $atomic_type, $int_type);
+ }
+ #[cfg(else)]
+ {
+ atomic_int!(cas[emulate], $atomic_type, $int_type);
+ }
+ });
+ // RMW (no-fetch)
+ #[cfg(not(target_arch = "msp430"))]
+ items!({
+ impl_default_no_fetch_ops!($atomic_type, $int_type);
+ impl $atomic_type {
+ #[inline]
+ pub(crate) fn not(&self, order: Ordering) {
+ self.fetch_not(order);
+ }
+ }
+ });
+ }
+ });
+ };
+ (all_critical_session, $atomic_type:ident, $int_type:ty, $align:literal) => {
+ atomic_int!(base, $atomic_type, $int_type, $align);
+ atomic_base!(emulate_load_store, $atomic_type, $int_type);
+ atomic_int!(cas[emulate], $atomic_type, $int_type);
+ impl_default_no_fetch_ops!($atomic_type, $int_type);
+ impl_default_bit_opts!($atomic_type, $int_type);
+ impl $atomic_type {
+ #[inline]
+ pub(crate) fn not(&self, order: Ordering) {
+ self.fetch_not(order);
+ }
}
- }
- };
- // RISC-V 8-bit/16-bit RMW with Zaamo extension
- (cas[sub_word], $atomic_type:ident, $int_type:ident) => {
- #[cfg(any(target_feature = "zabha", portable_atomic_target_feature = "zabha"))]
- atomic_int!(cas, $atomic_type, $int_type);
- #[cfg(not(any(target_feature = "zabha", portable_atomic_target_feature = "zabha")))]
- impl $atomic_type {
- #[inline]
- pub(crate) fn swap(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(val);
+ };
+ (emulate_arithmetic, $atomic_type:ident, $int_type:ty) => {
+ impl $atomic_type {
+ #[inline]
+ pub(crate) fn fetch_add(&self, val: $int_type, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev.wrapping_add(val), &guard);
prev
- })
- }
-
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn compare_exchange(
- &self,
- current: $int_type,
- new: $int_type,
- success: Ordering,
- failure: Ordering,
- ) -> Result<$int_type, $int_type> {
- crate::utils::assert_compare_exchange_ordering(success, failure);
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- if prev == current {
- self.v.get().write(new);
- Ok(prev)
- } else {
- Err(prev)
- }
- })
- }
-
- #[inline]
- #[cfg_attr(all(debug_assertions, not(portable_atomic_no_track_caller)), track_caller)]
- pub(crate) fn compare_exchange_weak(
- &self,
- current: $int_type,
- new: $int_type,
- success: Ordering,
- failure: Ordering,
- ) -> Result<$int_type, $int_type> {
- self.compare_exchange(current, new, success, failure)
- }
-
- #[inline]
- pub(crate) fn fetch_add(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev.wrapping_add(val));
+ }
+ #[inline]
+ pub(crate) fn fetch_sub(&self, val: $int_type, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev.wrapping_sub(val), &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_sub(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev.wrapping_sub(val));
+ }
+ #[inline]
+ pub(crate) fn fetch_max(&self, val: $int_type, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(core::cmp::max(prev, val), &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_and(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_and(val, order)
- }
-
- #[inline]
- pub(crate) fn fetch_nand(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(!(prev & val));
+ }
+ #[inline]
+ pub(crate) fn fetch_min(&self, val: $int_type, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(core::cmp::min(prev, val), &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_or(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_or(val, order)
- }
- #[inline]
- pub(crate) fn fetch_xor(&self, val: $int_type, order: Ordering) -> $int_type {
- self.as_native().fetch_xor(val, order)
+ }
}
-
- #[inline]
- pub(crate) fn fetch_max(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(core::cmp::max(prev, val));
+ };
+ (emulate_bit, $atomic_type:ident, $int_type:ty) => {
+ impl $atomic_type {
+ #[inline]
+ pub(crate) fn fetch_and(&self, val: $int_type, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev & val, &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_min(&self, val: $int_type, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(core::cmp::min(prev, val));
+ }
+ #[inline]
+ pub(crate) fn fetch_or(&self, val: $int_type, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev | val, &guard);
prev
- })
- }
-
- #[inline]
- pub(crate) fn fetch_not(&self, order: Ordering) -> $int_type {
- self.as_native().fetch_not(order)
- }
-
- #[inline]
- pub(crate) fn fetch_neg(&self, _order: Ordering) -> $int_type {
- // SAFETY: any data races are prevented by disabling interrupts (see
- // module-level comments) and the raw pointer is valid because we got it
- // from a reference.
- with(|| unsafe {
- let prev = self.v.get().read();
- self.v.get().write(prev.wrapping_neg());
+ }
+ #[inline]
+ pub(crate) fn fetch_xor(&self, val: $int_type, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(prev ^ val, &guard);
prev
- })
- }
- #[inline]
- pub(crate) fn neg(&self, order: Ordering) {
- self.fetch_neg(order);
+ }
+ #[inline]
+ pub(crate) fn fetch_not(&self, _order: Ordering) -> $int_type {
+ let guard = disable();
+ let prev = self.read(&guard);
+ self.write(!prev, &guard);
+ prev
+ }
}
- }
- };
-}
-
-#[cfg(target_pointer_width = "16")]
-#[cfg(not(target_arch = "avr"))]
-atomic_int!(load_store_atomic, AtomicIsize, isize, 2);
-#[cfg(target_pointer_width = "16")]
-#[cfg(not(target_arch = "avr"))]
-atomic_int!(load_store_atomic, AtomicUsize, usize, 2);
-#[cfg(target_arch = "avr")]
-atomic_int!(all_critical_session, AtomicIsize, isize, 2);
-#[cfg(target_arch = "avr")]
-atomic_int!(all_critical_session, AtomicUsize, usize, 2);
-#[cfg(target_pointer_width = "32")]
-atomic_int!(load_store_atomic, AtomicIsize, isize, 4);
-#[cfg(target_pointer_width = "32")]
-atomic_int!(load_store_atomic, AtomicUsize, usize, 4);
-#[cfg(target_pointer_width = "64")]
-atomic_int!(load_store_atomic, AtomicIsize, isize, 8);
-#[cfg(target_pointer_width = "64")]
-atomic_int!(load_store_atomic, AtomicUsize, usize, 8);
-#[cfg(target_pointer_width = "128")]
-atomic_int!(load_store_atomic, AtomicIsize, isize, 16);
-#[cfg(target_pointer_width = "128")]
-atomic_int!(load_store_atomic, AtomicUsize, usize, 16);
-
-#[cfg(not(all(target_arch = "avr", portable_atomic_no_asm)))]
-atomic_int!(load_store_atomic[sub_word], AtomicI8, i8, 1);
-#[cfg(not(all(target_arch = "avr", portable_atomic_no_asm)))]
-atomic_int!(load_store_atomic[sub_word], AtomicU8, u8, 1);
-#[cfg(all(target_arch = "avr", portable_atomic_no_asm))]
-atomic_int!(all_critical_session, AtomicI8, i8, 1);
-#[cfg(all(target_arch = "avr", portable_atomic_no_asm))]
-atomic_int!(all_critical_session, AtomicU8, u8, 1);
-#[cfg(not(target_arch = "avr"))]
-atomic_int!(load_store_atomic[sub_word], AtomicI16, i16, 2);
-#[cfg(not(target_arch = "avr"))]
-atomic_int!(load_store_atomic[sub_word], AtomicU16, u16, 2);
-#[cfg(target_arch = "avr")]
-atomic_int!(all_critical_session, AtomicI16, i16, 2);
-#[cfg(target_arch = "avr")]
-atomic_int!(all_critical_session, AtomicU16, u16, 2);
-
-#[cfg(not(target_pointer_width = "16"))]
-atomic_int!(load_store_atomic, AtomicI32, i32, 4);
-#[cfg(not(target_pointer_width = "16"))]
-atomic_int!(load_store_atomic, AtomicU32, u32, 4);
-#[cfg(target_pointer_width = "16")]
-#[cfg(any(test, feature = "fallback"))]
-atomic_int!(all_critical_session, AtomicI32, i32, 4);
-#[cfg(target_pointer_width = "16")]
-#[cfg(any(test, feature = "fallback"))]
-atomic_int!(all_critical_session, AtomicU32, u32, 4);
-
-cfg_has_fast_atomic_64! {
- atomic_int!(load_store_atomic, AtomicI64, i64, 8);
- atomic_int!(load_store_atomic, AtomicU64, u64, 8);
-}
-#[cfg(any(test, feature = "fallback"))]
-cfg_no_fast_atomic_64! {
- atomic_int!(all_critical_session, AtomicI64, i64, 8);
- atomic_int!(all_critical_session, AtomicU64, u64, 8);
-}
+ };
+ (cas[emulate], $atomic_type:ident, $int_type:ty) => {
+ atomic_base!(emulate_swap, $atomic_type, $int_type);
+ atomic_int!(emulate_arithmetic, $atomic_type, $int_type);
+ atomic_int!(emulate_bit, $atomic_type, $int_type);
+ };
+ // RISC-V 32-bit(RV32)/{32,64}-bit(RV64) RMW with Zaamo extension
+ // RISC-V 8-bit/16-bit RMW with Zabha extension
+ (cas, $atomic_type:ident, $int_type:ty) => {};
+ // RISC-V 8-bit/16-bit RMW with Zaamo extension
+ (cas[sub_word], $atomic_type:ident, $int_type:ty) => {
+ // RISC-V 8-bit/16-bit RMW with Zaamo+Zabha extension
+ #[cfg(any(target_feature = "zabha", portable_atomic_target_feature = "zabha"))]
+ atomic_int!(cas, $atomic_type, $int_type);
+
+ // RISC-V 8-bit/16-bit RMW with Zaamo extension
+ #[cfg(not(any(target_feature = "zabha", portable_atomic_target_feature = "zabha")))]
+ atomic_base!(emulate_swap, $atomic_type, $int_type);
+ #[cfg(not(any(target_feature = "zabha", portable_atomic_target_feature = "zabha")))]
+ atomic_int!(emulate_arithmetic, $atomic_type, $int_type);
+ };
+ }
-#[cfg(any(test, feature = "fallback"))]
-atomic_int!(all_critical_session, AtomicI128, i128, 16);
-#[cfg(any(test, feature = "fallback"))]
-atomic_int!(all_critical_session, AtomicU128, u128, 16);
+ #[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(
+ test,
+ target_arch = "avr",
+ target_arch = "msp430",
+ portable_atomic_no_atomic_cas,
+ ))
+ )]
+ #[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(
+ test,
+ target_arch = "avr",
+ target_arch = "msp430",
+ not(target_has_atomic = "ptr"),
+ ))
+ )]
+ items!({
+ // On some platforms, atomic load/store can be implemented in a more efficient
+ // way than disabling interrupts. On MSP430, some RMWs that do not return the
+ // previous value can also be optimized.
+ //
+ // Note: On single-core systems, it is okay to use critical session-based
+ // CAS together with atomic load/store. The load/store will not be
+ // called while interrupts are disabled, and since the load/store is
+ // atomic, it is not affected by interrupts even if interrupts are enabled.
+ #[cfg(not(any(
+ all(target_arch = "avr", portable_atomic_no_asm),
+ feature = "critical-section",
+ )))]
+ use self::arch::atomic;
+
+ #[cfg(target_pointer_width = "16")]
+ #[cfg(not(target_arch = "avr"))]
+ atomic_int!(load_store_atomic, AtomicIsize, isize, 2);
+ #[cfg(target_pointer_width = "16")]
+ #[cfg(not(target_arch = "avr"))]
+ atomic_int!(load_store_atomic, AtomicUsize, usize, 2);
+ #[cfg(target_arch = "avr")]
+ atomic_int!(all_critical_session, AtomicIsize, isize, 2);
+ #[cfg(target_arch = "avr")]
+ atomic_int!(all_critical_session, AtomicUsize, usize, 2);
+ #[cfg(target_pointer_width = "32")]
+ atomic_int!(load_store_atomic, AtomicIsize, isize, 4);
+ #[cfg(target_pointer_width = "32")]
+ atomic_int!(load_store_atomic, AtomicUsize, usize, 4);
+ #[cfg(target_pointer_width = "64")]
+ atomic_int!(load_store_atomic, AtomicIsize, isize, 8);
+ #[cfg(target_pointer_width = "64")]
+ atomic_int!(load_store_atomic, AtomicUsize, usize, 8);
+ #[cfg(target_pointer_width = "128")]
+ atomic_int!(load_store_atomic, AtomicIsize, isize, 16);
+ #[cfg(target_pointer_width = "128")]
+ atomic_int!(load_store_atomic, AtomicUsize, usize, 16);
+
+ #[cfg(not(all(target_arch = "avr", portable_atomic_no_asm)))]
+ atomic_int!(load_store_atomic[sub_word], AtomicI8, i8, 1);
+ #[cfg(not(all(target_arch = "avr", portable_atomic_no_asm)))]
+ atomic_int!(load_store_atomic[sub_word], AtomicU8, u8, 1);
+ #[cfg(all(target_arch = "avr", portable_atomic_no_asm))]
+ atomic_int!(all_critical_session, AtomicI8, i8, 1);
+ #[cfg(all(target_arch = "avr", portable_atomic_no_asm))]
+ atomic_int!(all_critical_session, AtomicU8, u8, 1);
+ #[cfg(not(target_arch = "avr"))]
+ atomic_int!(load_store_atomic[sub_word], AtomicI16, i16, 2);
+ #[cfg(not(target_arch = "avr"))]
+ atomic_int!(load_store_atomic[sub_word], AtomicU16, u16, 2);
+ #[cfg(target_arch = "avr")]
+ atomic_int!(all_critical_session, AtomicI16, i16, 2);
+ #[cfg(target_arch = "avr")]
+ atomic_int!(all_critical_session, AtomicU16, u16, 2);
+
+ #[cfg(not(target_pointer_width = "16"))]
+ atomic_int!(load_store_atomic, AtomicI32, i32, 4);
+ #[cfg(not(target_pointer_width = "16"))]
+ atomic_int!(load_store_atomic, AtomicU32, u32, 4);
+
+ cfg_has_fast_atomic_64! {
+ atomic_int!(load_store_atomic, AtomicI64, i64, 8);
+ atomic_int!(load_store_atomic, AtomicU64, u64, 8);
+ }
+ });
+
+ // Double or more width atomics (require fallback feature for consistency with other situations).
+ #[cfg(target_pointer_width = "16")]
+ #[cfg(any(test, feature = "fallback"))]
+ items!({
+ atomic_int!(all_critical_session, AtomicI32, i32, 4);
+ atomic_int!(all_critical_session, AtomicU32, u32, 4);
+ });
+ #[cfg(any(
+ test,
+ all(
+ feature = "fallback",
+ not(all(
+ target_arch = "riscv32",
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(
+ target_feature = "zacas",
+ portable_atomic_target_feature = "zacas",
+ all(
+ feature = "fallback",
+ not(portable_atomic_no_outline_atomics),
+ any(target_os = "linux", target_os = "android"),
+ ),
+ ),
+ )),
+ ),
+ ))]
+ #[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(test, portable_atomic_no_atomic_64))
+ )]
+ #[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(test, not(target_has_atomic = "64")))
+ )]
+ cfg_no_fast_atomic_64! {
+ atomic_int!(all_critical_session, AtomicI64, i64, 8);
+ atomic_int!(all_critical_session, AtomicU64, u64, 8);
+ }
+ #[cfg(any(
+ test,
+ all(
+ feature = "fallback",
+ not(all(
+ target_arch = "riscv64",
+ not(any(miri, portable_atomic_sanitize_thread)),
+ any(not(portable_atomic_no_asm), portable_atomic_unstable_asm),
+ any(
+ target_feature = "zacas",
+ portable_atomic_target_feature = "zacas",
+ all(
+ feature = "fallback",
+ not(portable_atomic_no_outline_atomics),
+ any(target_os = "linux", target_os = "android"),
+ ),
+ ),
+ )),
+ ),
+ ))]
+ items!({
+ atomic_int!(all_critical_session, AtomicI128, i128, 16);
+ atomic_int!(all_critical_session, AtomicU128, u128, 16);
+ });
+});
#[cfg(test)]
mod tests {
### external/vendor/portable-atomic/src/imp/interrupt/msp430.rs
@@ -9,32 +9,40 @@ Refs:
- MSP430x5xx and MSP430x6xx Family User's Guide, Rev. Q
https://www.ti.com/lit/ug/slau208q/slau208q.pdf
-Generated asm:
-- msp430 https://godbolt.org/z/fc6h89xac
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
#[cfg(not(portable_atomic_no_asm))]
use core::arch::asm;
+#[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(test, portable_atomic_no_atomic_cas))
+)]
+#[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(test, not(target_has_atomic = "ptr")))
+)]
pub(super) use super::super::msp430 as atomic;
-pub(super) type State = u16;
+pub(crate) type State = u16;
/// Disables interrupts and returns the previous interrupt state.
#[inline(always)]
-pub(super) fn disable() -> State {
+pub(crate) fn disable() -> State {
let sr: State;
// SAFETY: reading the status register and disabling interrupts are safe.
// (see module-level comments of interrupt/mod.rs on the safety of using privileged instructions)
+ //
+ // See "NOTE: Enable and Disable Interrupt" of User's Guide for NOP: https://www.ti.com/lit/ug/slau208q/slau208q.pdf#page=60
unsafe {
- // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
- // Do not use `preserves_flags` because DINT modifies the GIE (global interrupt enable) bit of the status register.
- // See "NOTE: Enable and Disable Interrupt" of User's Guide for NOP: https://www.ti.com/lit/ug/slau208q/slau208q.pdf#page=60
#[cfg(not(portable_atomic_no_asm))]
asm!(
"mov r2, {sr}", // sr = SR
"dint {{ nop", // SR.GIE = 0
sr = out(reg) sr,
+ // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
+ // Do not use `preserves_flags` because DINT modifies the GIE (global interrupt enable) bit of the status register.
options(nostack),
);
#[cfg(portable_atomic_no_asm)]
@@ -52,23 +60,24 @@ pub(super) fn disable() -> State {
///
/// The state must be the one retrieved by the previous `disable`.
#[inline(always)]
-pub(super) unsafe fn restore(prev_sr: State) {
+pub(crate) unsafe fn restore(prev_sr: State) {
// SAFETY: the caller must guarantee that the state was retrieved by the previous `disable`,
+ //
+ // This clobbers the entire status register, but we never explicitly modify
+ // flags within a critical session, and the only flags that may be changed
+ // within a critical session are the arithmetic flags that are changed as
+ // a side effect of arithmetic operations, etc., which LLVM recognizes,
+ // so it is safe to clobber them here.
+ // See also the discussion at https://github.com/taiki-e/portable-atomic/pull/40.
+ //
+ // See "NOTE: Enable and Disable Interrupt" of User's Guide for NOP: https://www.ti.com/lit/ug/slau208q/slau208q.pdf#page=60
unsafe {
- // This clobbers the entire status register, but we never explicitly modify
- // flags within a critical session, and the only flags that may be changed
- // within a critical session are the arithmetic flags that are changed as
- // a side effect of arithmetic operations, etc., which LLVM recognizes,
- // so it is safe to clobber them here.
- // See also the discussion at https://github.com/taiki-e/portable-atomic/pull/40.
- //
- // Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
- // Do not use `preserves_flags` because MOV modifies the status register.
- // See "NOTE: Enable and Disable Interrupt" of User's Guide for NOP: https://www.ti.com/lit/ug/slau208q/slau208q.pdf#page=60
#[cfg(not(portable_atomic_no_asm))]
asm!(
"nop {{ mov {prev_sr}, r2 {{ nop", // SR = prev_sr
prev_sr = in(reg) prev_sr,
+ // Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
+ // Do not use `preserves_flags` because MOV modifies the status register.
options(nostack),
);
#[cfg(portable_atomic_no_asm)]
### external/vendor/portable-atomic/src/imp/interrupt/riscv.rs
@@ -4,70 +4,77 @@
Refs:
- RISC-V Instruction Set Manual
Machine Status (mstatus and mstatush) Registers
- https://github.com/riscv/riscv-isa-manual/blob/riscv-isa-release-8b9dc50-2024-08-30/src/machine.adoc#machine-status-mstatus-and-mstatush-registers
+ https://github.com/riscv/riscv-isa-manual/blob/riscv-isa-release-56e76be-2025-08-26/src/machine.adoc#machine-status-mstatus-and-mstatush-registers
Supervisor Status (sstatus) Register
- https://github.com/riscv/riscv-isa-manual/blob/riscv-isa-release-8b9dc50-2024-08-30/src/supervisor.adoc#supervisor-status-sstatus-register
+ https://github.com/riscv/riscv-isa-manual/blob/riscv-isa-release-56e76be-2025-08-26/src/supervisor.adoc#supervisor-status-sstatus-register
See also src/imp/riscv.rs.
-Generated asm:
-- riscv64gc https://godbolt.org/z/zTrzT1Ee7
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
#[cfg(not(portable_atomic_no_asm))]
use core::arch::asm;
+#[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(test, portable_atomic_no_atomic_cas))
+)]
+#[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(test, not(target_has_atomic = "ptr")))
+)]
pub(super) use super::super::riscv as atomic;
-// Status register
-#[cfg(not(portable_atomic_s_mode))]
-macro_rules! status {
- () => {
- "mstatus"
- };
-}
-#[cfg(portable_atomic_s_mode)]
-macro_rules! status {
- () => {
- "sstatus"
- };
-}
-
-// MIE (Machine Interrupt Enable) bit (1 << 3)
-#[cfg(not(portable_atomic_s_mode))]
-const MASK: State = 0x8;
-#[cfg(not(portable_atomic_s_mode))]
-macro_rules! mask {
- () => {
- "0x8"
- };
-}
-// SIE (Supervisor Interrupt Enable) bit (1 << 1)
-#[cfg(portable_atomic_s_mode)]
-const MASK: State = 0x2;
-#[cfg(portable_atomic_s_mode)]
-macro_rules! mask {
- () => {
- "0x2"
- };
-}
+cfg_sel!({
+ // Supervisor-mode (S-mode)
+ #[cfg(portable_atomic_s_mode)]
+ {
+ // Status register
+ macro_rules! status {
+ () => {
+ "sstatus"
+ };
+ }
+ // SIE (Supervisor Interrupt Enable) bit (1 << 1)
+ #[cfg(portable_atomic_s_mode)]
+ macro_rules! mask {
+ () => {
+ "0x2"
+ };
+ }
+ }
+ // Machine-mode (M-mode)
+ #[cfg(else)]
+ {
+ // Status register
+ macro_rules! status {
+ () => {
+ "mstatus"
+ };
+ }
+ // MIE (Machine Interrupt Enable) bit (1 << 3)
+ macro_rules! mask {
+ () => {
+ "0x8"
+ };
+ }
+ }
+});
-#[cfg(target_arch = "riscv32")]
-pub(super) type State = u32;
-#[cfg(target_arch = "riscv64")]
-pub(super) type State = u64;
+pub(crate) type State = crate::utils::RegSize;
/// Disables interrupts and returns the previous interrupt state.
#[inline(always)]
-pub(super) fn disable() -> State {
+pub(crate) fn disable() -> State {
let status: State;
// SAFETY: reading mstatus/sstatus and disabling interrupts is safe.
// (see module-level comments of interrupt/mod.rs on the safety of using privileged instructions)
unsafe {
- // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
asm!(
concat!("csrrci {status}, ", status!(), ", ", mask!()), // atomic { status = status!(); status!() &= !mask!() }
status = out(reg) status,
+ // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
options(nostack, preserves_flags),
);
}
@@ -80,16 +87,16 @@ pub(super) fn disable() -> State {
///
/// The state must be the one retrieved by the previous `disable`.
#[inline(always)]
-pub(super) unsafe fn restore(status: State) {
- if status & MASK != 0 {
- // SAFETY: the caller must guarantee that the state was retrieved by the previous `disable`,
- // and we've checked that interrupts were enabled before disabling interrupts.
- unsafe {
+pub(crate) unsafe fn restore(prev_status: State) {
+ // SAFETY: the caller must guarantee that the state was retrieved by the previous `disable`,
+ //
+ // This clobbers the entire mstatus/sstatus register. See msp430.rs to safety on this.
+ unsafe {
+ asm!(
+ concat!("csrw ", status!(), ", {prev_status}"), // atomic { status!() = prev_status }
+ prev_status = in(reg) prev_status,
// Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
- asm!(
- concat!("csrsi ", status!(), ", ", mask!()), // atomic { status!() |= mask!() }
- options(nostack, preserves_flags),
- );
- }
+ options(nostack, preserves_flags),
+ );
}
}
### external/vendor/portable-atomic/src/imp/interrupt/xtensa.rs
@@ -5,26 +5,40 @@ Refs:
- Xtensa Instruction Set Architecture (ISA) Summary for all Xtensa LX Processors
https://www.cadence.com/content/dam/cadence-www/global/en_US/documents/tools/silicon-solutions/compute-ip/isa-summary.pdf
- Linux kernel's Xtensa atomic implementation
- https://github.com/torvalds/linux/blob/v6.11/arch/xtensa/include/asm/atomic.h
+ https://github.com/torvalds/linux/blob/v6.16/arch/xtensa/include/asm/atomic.h
+
+See tests/asm-test/asm/portable-atomic for generated assembly.
*/
use core::arch::asm;
-
+#[cfg_attr(
+ portable_atomic_no_cfg_target_has_atomic,
+ cfg(any(test, portable_atomic_no_atomic_cas))
+)]
+#[cfg_attr(
+ not(portable_atomic_no_cfg_target_has_atomic),
+ cfg(any(test, not(target_has_atomic = "ptr")))
+)]
pub(super) use core::sync::atomic;
-pub(super) type State = u32;
+pub(crate) type State = u32;
/// Disables interrupts and returns the previous interrupt state.
#[inline(always)]
-pub(super) fn disable() -> State {
+pub(crate) fn disable() -> State {
let ps: State;
// SAFETY: reading the PS special register and disabling all interrupts is safe.
// (see module-level comments of interrupt/mod.rs on the safety of using privileged instructions)
+ //
+ // Interrupt level 15 to disable all interrupts.
+ // SYNC after RSIL is not required.
unsafe {
- // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
- // Interrupt level 15 to disable all interrupts.
- // SYNC after RSIL is not required.
- asm!("rsil {0}, 15", out(reg) ps, options(nostack));
+ asm!(
+ "rsil {ps}, 15", // ps = PS; PS.INTLEVEL = 15
+ ps = out(reg) ps,
+ // Do not use `nomem` and `readonly` because prevent subsequent memory accesses from being reordered before interrupts are disabled.
+ options(nostack),
+ );
}
ps
}
@@ -35,16 +49,18 @@ pub(super) fn disable() -> State {
///
/// The state must be the one retrieved by the previous `disable`.
#[inline(always)]
-pub(super) unsafe fn restore(ps: State) {
+pub(crate) unsafe fn restore(prev_ps: State) {
// SAFETY: the caller must guarantee that the state was retrieved by the previous `disable`,
// and we've checked that interrupts were enabled before disabling interrupts.
+ //
+ // SYNC after WSR is required to guarantee that subsequent RSIL read the written value.
+ // See also 3.8.10 Processor Control Instructions of Xtensa Instruction Set Architecture (ISA) Summary for all Xtensa LX Processors.
unsafe {
- // Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
- // SYNC after WSR is required to guarantee that subsequent RSIL read the written value.
asm!(
- "wsr.ps {0}",
- "rsync",
- in(reg) ps,
+ "wsr.ps {prev_ps}", // PS = prev_ps
+ "rsync", // wait
+ prev_ps = in(reg) prev_ps,
+ // Do not use `nomem` and `readonly` because prevent preceding memory accesses from being reordered after interrupts are enabled.
options(nostack),
);
}
### external/vendor/portable-atomic/src/imp/mod.rs
[binary or diff unavailable]
### external/vendor/portable-atomic/src/imp/msp430.rs
[binary or diff unavailable]
### external/vendor/portable-atomic/src/imp/riscv.rs
[binary or diff unavailable]
### external/vendor/portable-atomic/src/imp/x86.rs
[binary or diff unavailable]
### external/vendor/portable-atomic/src/lib.rs
[binary or diff unavailable]
### external/vendor/portable-atomic/src/rustdoc.css
[binary or diff unavailable]
### external/vendor/portable-atomic/src/tests/helper.rs
[binary or diff unavailable]
### external/vendor/portable-atomic/src/tests/mod.rs
[binary or diff unavailable]
### external/vendor/portable-atomic/src/utils.rs
[binary or diff unavailable]
### external/vendor/portable-atomic/version.rs
[binary or diff unavailable]
### external/vendor/rtt-target/.cargo-checksum.json
[binary or diff unavailable]
### external/vendor/rtt-target/Cargo.toml
[binary or diff unavailable]
### external/vendor/rtt-target/src/defmt.rs
[binary or diff unavailable]
### external/vendor/rtt-target/src/init.rs
[binary or diff unavailable]
### external/vendor/rtt-target/src/lib.rs
[binary or diff unavailable]
### external/vendor/rtt-target/src/log.rs
[binary or diff unavailable]
### external/vendor/rtt-target/src/rtt.rs
[binary or diff unavailable]
### src/rust/.cargo/config.toml
[binary or diff unavailable]
### src/rust/Cargo.lock
[binary or diff unavailable]
### src/rust/Cargo.toml
[binary or diff unavailable]
### src/rust/bitbox-debug/Cargo.toml
[binary or diff unavailable]
### src/rust/bitbox-debug/src/lib.rs
[binary or diff unavailable]
### src/rust/util/Cargo.toml
[binary or diff unavailable]
### src/rust/util/src/log.rs
[binary or diff unavailable]Risk score
Our methodology →Why this scored 15/100
Human-validated context
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
No validated notes yet.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.