連結リストまみれでRustを学ぶ
何か問題を見つけましたか？または，最終的なコードを一気にチェックしたいですか？すべてGitHubにあります！
訳注: 日本語版のリポジトリはこちらです
注意: 本書の現行版は，rustc 1.31 (2018年12月8日) に初めてリリースされた Rust 2018 に基づいて書かれています． あなたの Rust ツールチェインが十分に新しい場合，
cargo newが生成するCargo.tomlファイルにはedition = "2018"(あるいは，あなたが遠い未来に読んでいるなら，もっと大きな数字！) という行が含まれているはずです． 古いツールチェインを利用することも可能ですが，それをすると秘密のハードモードが解除され， 本書の本文ではまったく触れられないコンパイルエラーが発生します．ウワー，楽しそう！
Rustで連結リストを実装するにはどうしたらいいか，という質問をよく受けます．正直なところ，答えはお客様のご要望によりますし，その場ですらすらお答えできるものではありません．そこで，そうした疑問をまとめて解決するためにこの本を書くことにしました．
これから６つの連結リストの実装を通して，Rust でのプログラミングの基礎と応用をまるごとお教えします．あなたが学んでいく内容は以下の通り：
- 次のポインタ型:
&,&mut,Box,Rc,Arc,*const,*mut,NonNull(?) - 所有権，借用，継承可変性, 内部可変性，コピー
- 次のキーワードすべて: 構造体，列挙型，
fn,pub,impl,use,... - パターンマッチング，ジェネリクス，デストラクタ
- テスト，新しいツールチェインのインストール，
miriの使用 - アンセーフなRust: 生ポインタ，エイリアス，借用のスタック, UnsafeCell, バリアンス
そう，連結リストというのは本当にやばいものなので，実装しようとするとこういった概念を全部扱わないといけないのです．
すべてサイドバーにありますが（モバイルなら折りたたまれているかもしれませんけど）ご参考までに簡単にこれから作るものを紹介します．
- A Bad Singly-Linked Stack
- An Ok Singly-Linked Stack
- A Persistent Singly-Linked Stack
- A Bad But Safe Doubly-Linked Deque
- An Unsafe Singly-Linked Queue
- TODO: An Ok Unsafe Doubly-Linked Deque
- Bonus: A Bunch of Silly Lists
同じ結果を得られるように，私はこれからターミナルで打つコマンドはすべて書き出すつもりです．また開発には Rust の標準パッケージマネージャである Cargo を使うことにします．Cargo を使わなくても Rust プログラムは書けますが，rustc を直接使うよりもずっといいです．ただいじくり回したいだけなら，Rust プレイグラウンドを使ってブラウザ上で実行することもできます．
この後の章では，rustup を使って追加の Rust ツールをインストールしていきます．Rust のツールチェインはすべてrustup を用いてインストールすることを強くお勧めします．
さっそくプロジェクトを作っていきましょう:
> cargo new --lib lists
> cd lists
各リストを別ファイルにまとめ，作業内容が失われないようにします．
次のことは言っておかねばならないでしょう: Rust を本当に学ぶつもりなら，コードを書き，コンパイラに怒られ，それが何を意味するかを理解するということは避けて通れません．皆さんにお見せするために，ふんだんにエラーを出すようにするつもりです．つよい Rust プログラマになるためには，Rust の優れたコンパイラエラーやドキュメントを読み，理解できるようになることがとても重要なのです．
ごめんなさい嘘をつきました．私が実際に遭遇したコンパイルエラーの数はこんなものではなく，ここでお見せするのは氷山の一角です．どの言語でも遭遇するようなコピー&ペーストのエラーなどは紹介していません．これはコンパイラに怒られるためのツアーなのです．
かなりスローペースで進めていくので，正直に言ってまじめな目的にはそぐわないでしょう．プログラミングは楽しくなきゃダメだと思うんです，いやほんとに．もしあなたがきまじめなタイプなら，情報がぱんぱんに詰まったお堅くてきっちりしたものを書いてほしいと思われるかもしれませんが，そんな人のために書いているわけではないんです．あなたは間違ってる．
注意事項
はっきり言っておきますが，私は連結リストが大嫌いです．吐き気を催すほど．連結リストはひどいデータ構造です．もちろん連結リストには以下のようなときには良い選択です:
- 大きなリストの分割や結合をたくさん，たくさん，たくさん！行いたいとき
- 同時並行で複数スレッドがロックせずに進行するようなものを書いているとき
- カーネルまたは埋め込みの開発を行っていて，intrusive list を使いたいとき
- 純粋な関数型言語を使っているとき．この場合はセマンティクスが限定されているしデータが不変なので，連結リストの扱いが簡単
しかし Rust プログラマにとってはこういったことはすべて超レアなケースです．99% のケースでは Vec (可変長配列，スタック) を使えば済みますし，残りの 1% のケースも VecDeque (両端キュー) で十分です．Vec や VecDeque は割り当ての頻度が少なく，メモリのオーバーヘッドも小さく，真のランダムアクセスとキャッシュの局所性を備えており，ほとんどの場合に適切なデータ構造です．
連結リストはトライ木と同じくらいニッチで，頼りないデータ構造です．トライ木はめったに使用されない構造で，平均的なプログラマが開発の中で使うことはないだろうと主張しても反論はほとんどないと思います．しかし連結リストは奇妙にも名前がよく知られています．学部生の全員が連結リストの書き方を教わりますものね．std::collections の中で私が唯一殺すことができなかったニッチなコレクションなんです．C++ のリストなんですよ！
私たちはコミュニティとして，連結リストを「標準的な」データ構造として認めるべきではないと思います．連結リストはいくつかの場合には素晴らしいデータ構造ですが，それは例外的なケースあって，一般的に有用というわけではありません．
この節の最初の段落を読んだだけで，この本を読むのをやめてしまうひとがいます．例えば，私が2段落目で示した連結リストの素晴らしい使用例の一つを挙げて，反論してこようとするのです．
私に対するいくつかの反論と，それに対する私からの再反論の詳細を以下に示しておきますが，Rust を学びたい方は第1章まで読み飛ばしてください．
パフォーマンスは必ずしも問題ではない
そうですね！ もしかしたら，あなたのアプリは I/O バウンドになっているのかもしれませんし， 問題のコードはめったに呼び出されないどうでもいい部分かもしれません． しかし，それは連結リストを使う理由にはなりません． これは，どんなものでも使うべきだという主張なのです． 連結リストにこだわる理由は何でしょう？ linked hash map (順序つきの辞書) を使えばいいじゃないですか！
パフォーマンスが問題でないのであれば，配列をそのまま使っても 全然 問題ないでしょう．
ポインタがそこにあれば，O(1) 時間で分割・追加・挿入・削除が可能
その通りですね！しかし Bjarne Stroustrup で指摘されているように，この利点は実際には重要ではありません． 配列のすべての要素をコピーするより，連結リストのポインタを取得する方がずっと多くの時間がかかるものだからです． 配列のコピーは実際には非常に高速です．
分割や結合のコストが作業負荷の大半を占めるような場合でない限り， キャッシュ効果やコードの複雑さによって 他のすべて の操作が受けるペナルティが，あらゆる理論的なメリットを上回ってしまいます．
しかしまぁ，分割や結合に多くの時間を費やすようなアプリを開発しているのであれば，連結リストにも利点があるかもしれませんね．
償却の余裕がない
かなりニッチな領域に足を踏み入れているようですね．ほとんどの場合は償却の余裕があります．
確かに，配列は 最悪の場合, 償却されます．
配列を使用しているからといって，コストも償却されているとは限りません．
もし格納する要素の数が予測できるなら（あるいは上限を決めておけば），必要なスペースをあらかじめ確保することができます．
私の経験では，必要な要素の数を予測できるのは ごく普通 のことです．
特に Rust では，すべてのイテレータがまさにこのような場合のために size_hint メソッドを持っています．
そうすると, push と pop はまさに O(1) 演算になります．
そして，連結リストの push と pop よりもかなり高速になります．
ポインタのオフセットを行い，バイトを書き込み，整数をインクリメントします．
アロケータの類を使う必要はありません．
低遅延はどうでしょうか？
しかし，そうですね，負荷が予測できない場合は，連結リストを使用することで最悪の場合の遅延を抑えることができるでしょうね！
連結リストはメモリ使用量が少ない
あー，これややこしいんですよね． 「標準的な」配列のリサイズ戦略は，配列の最大半分が空になるように拡大または縮小することです． これは実に多くのスペースを無駄にします． 特に Rust では，（また埋め戻すだけだともったいないから）コレクションを自動的に縮小しませんので， いくらでも多くのスペースが無駄になりえます．
But this is a worst-case scenario. In the best-case, an array stack only has three pointers of overhead for the entire array. Basically no overhead.
Linked lists on the other hand unconditionally waste space per element. A singly-linked list wastes one pointer while a doubly-linked list wastes two. Unlike an array, the relative wasteage is proportional to the size of the element. If you have huge elements this approaches 0 waste. If you have tiny elements (say, bytes), then this can be as much as 16x memory overhead (8x on 32-bit)!
Actually, it's more like 23x (11x on 32-bit) because padding will be added to the byte to align the whole node's size to a pointer.
This is also assuming the best-case for your allocator: that allocating and deallocating nodes is being done densely and you're not losing memory to fragmentation.
But yes, if you have huge elements, can't predict your load, and have a decent allocator, there are memory savings to be had!
I use linked lists all the time in <functional language>
Great! Linked lists are super elegant to use in functional languages because you can manipulate them without any mutation, can describe them recursively, and also work with infinite lists due to the magic of laziness.
Specifically, linked lists are nice because they represent an iteration without the need for any mutable state. The next step is just visiting the next sublist.
Rust mostly does this kind of thing with iterators. They can be infinite and you can map, filter, reverse, and concatenate them just like a functional list, and it will all be done just as lazily!
Rust also lets you easily talk about sub-arrays with slices. Your usual
head/tail split in a functional language is just slice.split_at_mut(1).
For a long time, Rust had an experimental system for pattern matching on
slices which was super cool, but the feature was simplified when it was
stabilized. Still, basic slice patterns are neat! And of course,
slices can be turned into iterators!
But yes, if you're limited to immutable semantics, linked lists can be very nice.
Note that I'm not saying that functional programming is necessarily weak or bad. However it is fundamentally semantically limited: you're largely only allowed to talk about how things are, and not how they should be done. This is actually a feature, because it enables the compiler to do tons of exotic transformations and potentially figure out the best way to do things without you having to worry about it. However this comes at the cost of being able to worry about it. There are usually escape hatches, but at some limit you're just writing procedural code again.
Even in functional languages, you should endeavour to use the appropriate data structure for the job when you actually need a data structure. Yes, singly-linked lists are your primary tool for control flow, but they're a really poor way to actually store a bunch of data and query it.
Linked lists are great for building concurrent data structures!
Yes! Although writing a concurrent data structure is really a whole different beast, and isn't something that should be taken lightly. Certainly not something many people will even consider doing. Once one's been written, you're also not really choosing to use a linked list. You're choosing to use an MPSC queue or whatever. The implementation strategy is pretty far removed in this case!
But yes, linked lists are the defacto heroes of the dark world of lock-free concurrency.
Mumble mumble kernel embedded something something intrusive.
It's niche. You're talking about a situation where you're not even using your language's runtime. Is that not a red flag that you're doing something strange?
It's also wildly unsafe.
But sure. Build your awesome zero-allocation lists on the stack.
Iterators don't get invalidated by unrelated insertions/removals
That's a delicate dance you're playing. Especially if you don't have a garbage collector. I might argue that your control flow and ownership patterns are probably a bit too tangled, depending on the details.
But yes, you can do some really cool crazy stuff with cursors.
They're simple and great for teaching!
Well, yeah. You're reading a book dedicated to that premise. Well, singly-linked lists are pretty simple. Doubly-linked lists can get kinda gnarly, as we'll see.
Take a Breath
Ok. That's out of the way. Let's write a bajillion linked lists.
単方向リストのだめな実装
Rust 言語の基本的な部分をすべて紹介し，よく理解するために「難しい方法」で作っていくという方針をとったため，この章はとても長くなりました．
最初のリストは src/first.rs に置きます．Rust に first.rs が私たちのライブラリを使用するファイルだということを知らせる必要がありますが，
そのためには (Cargo が作ってくれた) src/lib.rs の最初の行に次のように記述します．
// in lib.rs
pub mod first;
基本的なデータ設計
さて，連結リストとは何でしょうか．基本的にはヒープ (カーネルのひとたち，静かに！) 上にある， 順番にお互いを指すデータの断片の束のことです．連結リストは手続き型のプログラマにとっては 長い棒でつんつんするのも避けたい代物ですが，関数型のプログラマはあらゆることに使います． それでは関数型プログラマに連結リストの定義を訊いてみましょう．きっとこんな返事が返ってくると思います:
List a = Empty | Elem a (List a)
これは，おおよそ「リストは空か，リストに続く要素のどちらかである」と読めます．これは再帰的な定義を
タグ付き共用型 (sum type, tagged union type) を使って表したものです．ただし，タグ付き共用型というのは
「異なる型の値を持つことができる型」というのをおしゃれに言ったものです．Rust ではこれを enum (列挙型)
と呼んでいます．あなたが C 系の言語をすでにご存じなら，話が早いかもしれません．Rust の enum は，あなたが
知っている（そして愛している）enum と同じものです．では，先ほどの関数型定義を Rust で書き直してみましょう．
今のところ，シンプルにするためジェネリクスは使わないでおきます．符号付き 32 ビット整数 i32 のみを格納することにします．
// in first.rs
// このモジュールを外部のひとが使えるように pub を付けています
pub enum List {
Empty,
Elem(i32, List),
}
ふー，忙しい忙しい．さっさとコンパイルしちゃいましょう．
> cargo build
error[E0072]: recursive type `first::List` has infinite size
--> src/first.rs:4:1
|
4 | pub enum List {
| ^^^^^^^^^^^^^ recursive type has infinite size
5 | Empty,
6 | Elem(i32, List),
| ---- recursive without indirection
|
= help: insert indirection (e.g., a `Box`, `Rc`, or `&`) at some point to make `first::List` representable
なんだよ，関数型プログラマに騙された！（個人の意見です）
(裏切り行為を乗り越えた後) 実際にエラーメッセージを確認すると，実は rustc がこの問題を解決する方法を 正確に教えてくれていることがわかります．
first::Listを表すために，どこかでBox,Rc,&などの間接参照を用いてください
さてさて，Box ってなんでしょうね？ Rust box でググってみよう...．
どれどれ...．
pub struct Box<T>(_);ヒープ割り当てのためのポインタ型．詳しくは，モジュールごとのドキュメントを参照してください．
リンクをクリック．
Box<T>(通称box) は，Rust で最も単純なヒープ割り当ての形式です．ボックスは，この割り当ての所有権を持ち，ボックスがスコープを抜けるとその内容が破棄されます．例
ボックスを作る:
let x = Box::new(5);再帰的なデータ構造を作る:
#![allow(unused)] fn main() { #[derive(Debug)] enum List<T> { Cons(T, Box<List<T>>), Nil, } }
fn main() {
let list: List<i32> = List::Cons(1, Box::new(List::Cons(2, Box::new(List::Nil))));
println!("{:?}", list);
}
上のコードは
Cons(1, Box(Cons(2, Box(Nil))))と出力します．次のような形の再帰的な定義はボックス化する必要があります．
Cons(T, List<T>),そのままでは上手くいきません．なぜなら，リストのサイズがリスト自身に含まれている要素の数に依存するので，
Consに割り当てるべきメモリの量がわからないからです．ボックスはサイズが決まっているので，Consに 必要な大きさを事前に知ることができます．
ワーオ．こんなに適切で役に立つドキュメントは見たことがありませんね．私たちが書いたものがなぜ上手くいかなかったのか， そしてどう修正すれば良いのかがずばり書かれています．
やっぱりドキュメントは最高だな！
よし，やってみよう．
pub enum List {
Empty,
Elem(i32, Box<List>),
}
> cargo build
Finished dev [unoptimized + debuginfo] target(s) in 0.22s
やった，ビルドが通ったぞ！
しかし，これはいくつかの理由から，実に愚かな定義です．
2つの要素を持つリストを考えてみましょう:
[] = スタック
() = ヒープ
[Elem A, ptr] -> (Elem B, ptr) -> (Empty, *junk*)
重要な問題が2つあります．
- 「ぼくはノードじゃない」というだけのノードを割り当てている．
- ノードの一つに全くヒープが確保されていない．
表面的には，この2つは互いに打ち消しあうように見えます．余分なノードをヒープに割り当てる一方で， ノードの１つはヒープではなくスタックに割り当てています．しかし次のような設計のリストを考えてみてください．
[ptr] -> (Elem A, ptr) -> (Elem B, *null*)
この設計では，ノードは常にヒープに割り当てています．最初の設計との重要な違いは，ジャンクがないことです．
このジャンクとは何でしょうか？それを理解するために，enum がどのようにメモリ上に配置される
かを見る必要があります．
一般的に，次のような enum があったとします:
enum Foo {
D1(T1),
D2(T2),
...
Dn(Tn),
}
Foo は，それがどの列挙型のヴァリアント (D1, D2, .. Dn) を表しているかを示す整数を格納する必要があります．
これを列挙型のタグといいます．また，T1, T2, .. Tn のうち最大のものを格納できるだけのスペースも必要です．
(さらに，アラインメントの要件を満たすための余分なスペースも必要)．
ここで重要なのは，Empty が1ビットの情報であっても，いつでも Elem になれるように準備しなければならないので，
ポインタと要素のために必要なスペースを必ず消費する，ということです．したがって，最初の設計ではヒープにジャンクでいっぱいの
余分な要素が割り当てられ，2つめの設計よりも少し多くのスペースを消費してしまうのです．
ノードの一つをヒープに割り当てていないことも，悪いことです．なんでも常にヒープに割り当てる方がましです．これはノードの設計が不均一になるからです． ノードのプッシュやポップにはあまり影響しませんが，リストの分割やマージには影響します．
リストの分割を行ったときにどうなるか，両方の設計を比較してみましょう:
設計 1:
[Elem A, ptr] -> (Elem B, ptr) -> (Elem C, ptr) -> (Empty *junk*)
C で分割すると:
[Elem A, ptr] -> (Elem B, ptr) -> (Empty *junk*)
[Elem C, ptr] -> (Empty *junk*)
設計 2:
[ptr] -> (Elem A, ptr) -> (Elem B, ptr) -> (Elem C, *null*)
C で分割すると:
[ptr] -> (Elem A, ptr) -> (Elem B, *null*)
[ptr] -> (Elem C, *null*)
設計2の分割では，Bのポインタをスタックにコピーし，古いポインタを null にするだけです．
設計1では同じことをするために，さらにCをヒープからスタックにコピーする必要があります．
逆になるだけで，マージの場合も同様です．
ノード自体に要素を構築し，それを動かすことなくリスト間で自由に並び替えることが できる点が，連結リストの数少ない長所でした．ポインターをいじるだけで，ものが「移動」するのです． 設計1ではこの性質がゴミと化しています．
なるほど設計1がよくないことはわかりました．ではどのように書き換えればいいのでしょうか？ そうですね，次のような方法はどうでしょう:
pub enum List {
Empty,
ElemThenEmpty(i32),
ElemThenNotEmpty(i32, Box<List>),
}
これが更に悪いアイデアに見えることを祈ります．なぜなら，このリストには ElemThenNotEmpty(0, Box(Empty)) という完全に無効な状態が存在するからです．
また，要素の割り当てが依然として一様でないことも問題です．
しかし，この設計には1つ興味深い特徴があります．Empty ケースの割り当てを完全に回避し，ヒープ割り当ての総数を1つ減らすことができるのです． 残念なことに，これによって以前の設計で利用できていた ヌルポインタ最適化 が利用できなくなるので，さらにスペースを浪費する結果になってしまうのですが．
以前，すべての列挙型は，そのビットがどの列挙型のヴァリアントを表しているのかを指定するタグを格納しなければならないとご説明しました． しかし，その例外となる特別な種類の列挙型があります:
enum Foo {
A,
B(ContainsANonNullPtr),
}
この場合，ヌルポインタ最適化が行われ，タグにスペースを割くことがなくなります．
もしヴァリアントが A ならば，enum 全体がすべて 0 に設定されます．そうでなければヴァリアントは B です．
これが上手くいくのは, B はゼロでないポインタを含んでいるので，決してゼロにはならないからです．巧妙ですね．
他の列挙型や型でもこのような最適化ができるでしょうか？実はこういうケースはたくさんあります！
Rust が enum の設計を全く指定しないのはこのためです．
Rust はもっと複雑な enum 設計の最適化も行ってくれますが，最も重要なのは間違いなくヌルポインタ最適化です．
これは，&, &mut, Box, Rc, Arc, Vec, その他 Rust の重要な型が Option に入れられたとき，
オーバーヘッドがないことを意味します！ (後で一部を紹介します)
では，どのようにすれば余分なジャンクを発生させず，割り当ても均一にし，ヌルポインタ最適化も 行うということができるでしょうか？要素を持つことと，別のリストを割り当てることとをうまく切り離す 必要があります．そのためには，もう少しC言語的に考える必要があります．構造体です！
列挙型では複数の値のいずれか ひとつ を格納できましたが，構造体では一度に 多くの 値を格納できます． List を「リスト」と「ノード」の２つの型に分割してみましょう．
前述したように，List は Empty (空) であるか，ある要素の後に別の List が続くかのどちらかです．
この「要素があり，後に別のリストが続く」ケースを全く別の型で表現することで，Box を最適な位置にもっていくことができます．
struct Node {
elem: i32,
next: List,
}
pub enum List {
Empty,
More(Box<Node>),
}
チェックリストを確認していきましょう:
- リストの末尾に余分なジャンクを割り当てない: OK ✅
enumのおいしいヌルポインタ最適化を利用できる: OK ✅- すべての要素が均一に割り当てられている: OK ✅
ヨシ！ 実はたったいま構築したのは，Rust の公式ドキュメントで提案されている設計が よくないことを示すために最初に使用した設計そのものです．
> cargo build
warning: private type `first::Node` in public interface (error E0446)
--> src/first.rs:8:10
|
8 | More(Box<Node>),
| ^^^^^^^^^
|
= note: #[warn(private_in_public)] on by default
= warning: this was previously accepted by the compiler but
is being phased out; it will become a hard error in a future release!
:(
また Rust が怒ってますね．List を (みんなが使えるように) パブリックにしたのですが，Node はパブリックにしませんでした．
List の内部は完全にパブリックなので，その内部でプライベートな型について話すのは許されない，ということのようです．
Node 全体をパブリックにすることもできますが，Rust では一般に実装の詳細をプライベートにしておくことが推奨されます．
そこで，List を構造体にして，実装の詳細は隠しておけるようにしましょう．
pub struct List {
head: Link,
}
enum Link {
Empty,
More(Box<Node>),
}
struct Node {
elem: i32,
next: Link,
}
構造体 List はフィールドを１つしか持たないので，そのサイズはフィールドのサイズと同じです．ゼロコスト抽象化です！やったね．
> cargo build
warning: field is never used: `head`
--> src/first.rs:2:5
|
2 | head: Link,
| ^^^^^^^^^^
|
= note: #[warn(dead_code)] on by default
warning: variant is never constructed: `Empty`
--> src/first.rs:6:5
|
6 | Empty,
| ^^^^^
warning: variant is never constructed: `More`
--> src/first.rs:7:5
|
7 | More(Box<Node>),
| ^^^^^^^^^^^^^^^
warning: field is never used: `elem`
--> src/first.rs:11:5
|
11 | elem: i32,
| ^^^^^^^^^
warning: field is never used: `next`
--> src/first.rs:12:5
|
12 | next: Link,
| ^^^^^^^^^^
よし，コンパイルも通りました．Rust がだいぶ怒ってますが，これは Rust の知る限り
私たちが書いたものが全く役に立たないからです．
私たちは head を使用していませんし，私たちのライブラリを使う人もプライベートなので使うことができません．
ということは，Link と Node も役に立たないということになります．では，それを解決しましょう！
次は，いま作った私たちの List のためのコードを実装していきます．
New メソッド
実際のコードを型に関連付けるには，impl ブロックを使用します:
impl List {
// TODO, コードを書く
}
後は，実際にどんなコードを書くか考えるだけです．Rust では，次のように関数を宣言します．
fn foo(arg1: Type1, arg2: Type2) -> ReturnType {
// 関数の中で行う処理
}
最初に欲しいのは，リストを新しく作る方法です．実装の詳細は隠しておきたいので，
それを関数として提供する必要があります．Rust では，こういう場合には静的メソッドを使用するのが一般的です．
静的メソッドは，impl ブロックの中にある普通の関数です:
impl List {
pub fn new() -> Self {
List { head: Link::Empty }
}
}
ここで注釈を少し:
Selfは「implの横の，いちばん上に書いたあの型」の別名です．これで同じコードを繰り返さずに済みます．- 構造体のインスタンスは，構造体を宣言するのとほぼ同じ方法で作成できますが，フィールドの型を指定する代わりに，値を与えて初期化します．
enumのヴァリアントは，名前空間演算子である::を使って参照します．- 関数は暗黙のうちに最後の式を返します．これにより単純な関数が少しすっきりします．他の C 系の言語と同様に，
returnを使って関数の終端に来る前に値を返すことも可能です．
所有権のいろは
さて，リストを作ることができたので，それを使って何かできたらいいですね．
そこで「通常の」(静的ではない) メソッドを使用します．
メソッドは Rust の関数の中でも特殊で，self という型を持たない引数を取ります:
fn foo(self, arg2: Type2) -> ReturnType {
// 関数の中で行う処理
}
self には，self, &mut self, &self という3つの主要な形があります．
この3つの形態は，それぞれ Rust における所有権の3つの主要な形態に対応しています:
self- 値&mut self- 可変参照&self- 共有参照
値は，真の 所有権を表します．移動，破壊，変更，参照による貸し出しなど，値に対してやりたいことは何でもできます．値として何かを渡すと，その値は新しい場所に ムーブ されます．
新しい場所がその値を所有することになるので，古い場所はもうその値にアクセスできなくなります．
このため，ほとんどのメソッドは self を必要としません．
リストを一度使ったら，もう使えないというのではかなりイライラしますね．
可変参照は，自分が所有していない値への一時的で排他的なアクセスを表します．
可変参照している値に対しては，終了時に有効な状態で残しておけば，何をやっても許されます．
(無効にしてしまったら，所有者に対して失礼です！)
つまり，完全に値を上書きすることもできます．
ある値を別の値に入れ替えることができるので，これは本当に便利です．私たちも後でたくさん使うことになるでしょう．
&mut でできないことは，置換せずに値をムーブすることだけです．
&mut self は，self を変更したいメソッドにぴったりです．
共有参照は，自分が所有していない値への一時的な共有アクセスを表します．
共有アクセスなので，一般的には何も変更することができません．
これは，値を博物館に展示するようなものだと考えてください．
&self は，self を観察するだけのメソッドに最適です．
後に，値の変更に関するルールを迂回できるケースがあることを紹介します．
共有参照が 不変参照 と呼ばれないのはこのためです．
本来なら可変参照も 一意参照 (訳注: 原文は unique references) と呼んだ方がいいかもしれませんが，所有権と変性 (訳注: 原文は mutability で，値が変更できるかどうかという性質) を結びつけることで 99% のケースで直感的にわかりやすくなります．
Push メソッド
リストに値をプッシュする操作を実装していきましょう．
push はリストを変更するので，&mut self を引数に取ります．
さらに，プッシュする値が必要なので i32 型の変数も引数に用意します．
impl List {
pub fn push(&mut self, elem: i32) {
// TODO
}
}
まず最初に，要素を格納するためのノードを作ります:
pub fn push(&mut self, elem: i32) {
let new_node = Node {
elem: elem,
next: ?????
};
}
next の中身はなんでしょう? 以前のリスト全体ですね！
これで…できるかな？
impl List {
pub fn push(&mut self, elem: i32) {
let new_node = Node {
elem: elem,
next: self.head,
};
}
}
> cargo build
error[E0507]: cannot move out of borrowed content
--> src/first.rs:19:19
|
19 | next: self.head,
| ^^^^^^^^^ cannot move out of borrowed content
あーもう！Rust は正しいことを言ってるんでしょうが，具体的に何を意味しているのか， どうすればいいのかわかりません:
cannot move out of borrowed content
借用中の値をムーブすることはできません
私たちは今 self.head フィールドを next にムーブしようとしましたが，それが Rust のお咎めを受けました．
これでは，借用を終了して正当な所有者に「返す」ときに，部分的にしか初期化されていない self を返すことになります．
先に述べたように，&mut でこんなことはできません．
Rust は非常に礼儀正しいからです．こんな失礼なことはあってはなりません．
(びっくりするほど危険なことでもありますが，礼を失することに比べればとるに足りないことです)．
ムーブされた部分を埋めてみたらどうでしょうか？つまり，いま作ったリストを返すわけです:
pub fn push(&mut self, elem: i32) {
let new_node = Box::new(Node {
elem: elem,
next: self.head,
});
self.head = Link::More(new_node);
}
> cargo build
error[E0507]: cannot move out of borrowed content
--> src/first.rs:19:19
|
19 | next: self.head,
| ^^^^^^^^^ cannot move out of borrowed content
ダメでした．実際には Rust はこれを受け入れてもよいのですが，（様々な理由 -- 最も深刻なのは例外安全性です -- により）
受け容れません．
Rust に気づかれないように，head を取得する方法が必要なのです．
そこで，悪名高き Rust ハッカー，インディアナ・ジョーンズに助言を求めましょう:
おっ，インディは mem::replace を使うことを提案していますね．
この信じられないほど便利な関数を使えば，値を別の値で 置き換える ことで，借りた値を盗むことができます．
mem を使用するためには，ファイルの先頭に std::mem を引っ張ってきたうえで:
use std::mem;
使うべきところで使えばよいです:
pub fn push(&mut self, elem: i32) {
let new_node = Box::new(Node {
elem: elem,
next: mem::replace(&mut self.head, Link::Empty),
});
self.head = Link::More(new_node);
}
ここでは self.head を一時的に Link::Empty に replace してから，リストの新しい head に置き換えています．
率直に言って，これはかなり不満の残るコードです．しかし悲しいかな，そうせざるを得ないのですよ．
今のところはね．
でも，これで push メソッドが完成しました！たぶんだけど．
正直，テストをしないとわかりません．
今テストをする最も簡単な方法は，おそらく pop も書いて，正しい結果が得られるか確認することでしょう．
Pop メソッド
push と同様，pop はリストを変更するメソッドです．
push と異なるところは，実際に何かしらの値を返すことですね．
pop はリストが空だったらどうするかという厄介なコーナーケースも扱う必要があります．
このケースに対処するために，Option 型という頼もしい道具を使用します:
pub fn pop(&mut self) -> Option<i32> {
// TODO
}
Option<T> は存在するかどうかわからない値を扱う列挙型です．
Some(T) か None かのいずれかの値を取ります．
Link のときにやったように独自の enum を作ってもいいのですが，ユーザに戻り値が何なのか理解してほしいので，
どこにでもあり 誰もが 知っている Option 型を使うのが得策です．
実際，Option は非常に基本的なものなので，すべてのファイルで暗黙のうちにスコープに取り込まれます．
そのヴァリアントである Some や None も同様です．
(ですから，Option::None などと書く必要はないのです)
Option<T> の <T> は，Option が T に対するジェネリックであることを示しています．
つまり，どんな型に対しても Option を作ることができるのです！
さて，この Link ですが，Empty か More のどちらなのか，どうやって判断すればいいでしょう?
match でパターンマッチしてみますか！
pub fn pop(&mut self) -> Option<i32> {
match self.head {
Link::Empty => {
// TODO
}
Link::More(node) => {
// TODO
}
};
}
> cargo build
error[E0308]: mismatched types
--> src/first.rs:27:30
|
27 | pub fn pop(&mut self) -> Option<i32> {
| --- ^^^^^^^^^^^ expected enum `std::option::Option`, found ()
| |
| this function's body doesn't return
|
= note: expected type `std::option::Option<i32>`
found type `()`
おっと，pop は値を返さないといけないのに，まだやっていませんでしたね．
None を返すこともできますが，この場合，
関数の実装が終わっていないことを示すために unimplemented!() を返す方が良いでしょう．
unimplemented!() はマクロで (! はマクロを表します)，これに到達するとプログラムがパニックします．
(パニックとは，制御された方法でプログラムをクラッシュさせること)．
pub fn pop(&mut self) -> Option<i32> {
match self.head {
Link::Empty => {
// TODO
}
Link::More(node) => {
// TODO
}
};
unimplemented!()
}
無条件のパニックは，発散する関数 (diverging function) の例です．
発散する関数は呼び出し元に戻ることがないので，本来期待される値の型がなんであったとしても，
使用することができます．
ここでは，Option<T> 型の代わりに unimplemented!() を使用しています．
また，プログラム中に return を書く必要はないことに注意してください．
関数の最後の式 (基本的には行) が，暗黙のうちに戻り値になります．
これによって，すごく単純な関数をさらに単純にすることができます．
他の C 系の言語と同様に，return で早めに値を明示的に返すこともできます．
> cargo build
error[E0507]: cannot move out of borrowed content
--> src/first.rs:28:15
|
28 | match self.head {
| ^^^^^^^^^
| |
| cannot move out of borrowed content
| help: consider borrowing here: `&self.head`
...
32 | Link::More(node) => {
| ---- data moved here
|
note: move occurs because `node` has type `std::boxed::Box<first::Node>`, which does not implement the `Copy` trait
--> src/first.rs:32:24
|
32 | Link::More(node) => {
| ^^^^
おい Rust, 俺たちの邪魔をするな！
いつものように，Rust がカンカンに起こっています．
でも今回はちゃんと理由を教えてくれました．
デフォルトでは，パターンマッチは中身を新しいブランチにムーブしようとしますが，
ここでは self を値として所有していないため，これが行えないのです．
help: consider borrowing here: `&self.head`
ヘルプ: 借用することを検討してみてください: `&self.head`
Rust が言うには，match を参照にすれば直るみたいですね．🤷♀️
試してみましょう:
pub fn pop(&mut self) -> Option<i32> {
match &self.head {
Link::Empty => {
// TODO
}
Link::More(node) => {
// TODO
}
};
unimplemented!()
}
> cargo build
warning: unused variable: `node`
--> src/first.rs:32:24
|
32 | Link::More(node) => {
| ^^^^ help: consider prefixing with an underscore: `_node`
|
= note: #[warn(unused_variables)] on by default
warning: field is never used: `elem`
--> src/first.rs:13:5
|
13 | elem: i32,
| ^^^^^^^^^
|
= note: #[warn(dead_code)] on by default
warning: field is never used: `next`
--> src/first.rs:14:5
|
14 | next: Link,
| ^^^^^^^^^^
万歳，コンパイルが通ったぞ！
それでは，TODO で残しておいたロジック部分を考えていきましょう．
Option を作りたいので，そのための変数を作ります．
Empty の場合には None を返さないといけませんね．
More の場合には Some(i32) を返して，リストの先頭を変更する必要があります．
おおまかにはこんな感じでどうでしょうか？
pub fn pop(&mut self) -> Option<i32> {
let result;
match &self.head {
Link::Empty => {
result = None;
}
Link::More(node) => {
result = Some(node.elem);
self.head = node.next;
}
};
result
}
> cargo build
Compiling lists v0.1.0 (/Users/ABeingessner/dev/temp/lists)
error[E0507]: cannot move out of borrowed content
--> src/first.rs:35:29
|
35 | self.head = node.next;
| ^^^^^^^^^ cannot move out of borrowed content
ゴン！(頭を机に叩きつけた音)
共有参照しているものを，node からムーブしようとしてしまったようです．
立ち止まって，これからやろうとしていることが何なのか考え直すべきでしょう． 私たちがやりたいのは次のようなことです:
- リストが空かどうかチェックする
- 空であれば，
Noneを返せばいいです - 空でなかったときは以下を実行します:
- リストの先頭を削除する
elemを削除する- リストの先頭を
nextで置き換える Some(elem)を返す
重要なことは，私たちが 削除 をしようとしているという点です．
これはつまるところリストの先頭を（所有権のある）値として取得する必要があることを意味します．
共有参照である &self.head では，そんなことはできません．
さらに self への参照は既にある可変参照「だけ」なので，ムーブするには置き換えるしかありません．
どうやら，また我々は Empty ダンスをしているようです！
試してみましょう:
pub fn pop(&mut self) -> Option<i32> {
let result;
match mem::replace(&mut self.head, Link::Empty) {
Link::Empty => {
result = None;
}
Link::More(node) => {
result = Some(node.elem);
self.head = node.next;
}
};
result
}
cargo build
Finished dev [unoptimized + debuginfo] target(s) in 0.22s
ありえない…．
警告を ひとつも 出さずにコンパイルできました!!!!!
個人的なこだわりですが，ここでコード整形をしましょう: result という変数を使いましたが，実はこれは不要です．
関数が最後の式で評価されるのと同じように，match の各ブロックも最後の式で評価されます．
通常はセミコロンを付けて文にすることでこの挙動を抑制し，ブロックの評価を空のタプル () にしますね．
この () は push のように戻り値を宣言しない関数の返り値です．
そこで，代わりに pop をこう書くことができます:
pub fn pop(&mut self) -> Option<i32> {
match mem::replace(&mut self.head, Link::Empty) {
Link::Empty => None,
Link::More(node) => {
self.head = node.next;
Some(node.elem)
}
}
}
より簡潔で慣用的なコードになりました．
Link::Empty ブランチでは，中括弧 {} が完全に失われていることに注意してください．
評価する式が１つしかないときには省略できるんです．
cargo build
Finished dev [unoptimized + debuginfo] target(s) in 0.22s
ちゃんとコンパイルも通りました．いいね！
テスト
さて，push と pop を書き終えたので，次は書いたコードを実際にテストしてみましょう！
Rust と cargo は，ファーストクラスの機能としてテストをサポートしているので，これは非常に簡単です．
関数を書いて，#[test] というアノテーション (注釈) を付けるだけでいいんです．
一般的に，Rust コミュニティではテストはテストされるコードの隣におくのが慣習です．
しかし「本当の」コードとの衝突を避けるために，通常テスト用に新しい名前空間を作ります．
first.rs が lib.rs に含まれるよう指定するために mod を使いましたが，
mod はインラインで全く新しいファイルを作成することにも使用することができます．
// in first.rs
mod test {
#[test]
fn basics() {
// TODO
}
}
cargo test コマンドで呼び出します.
> cargo test
Compiling lists v0.1.0 (/Users/ABeingessner/dev/temp/lists)
Finished dev [unoptimized + debuginfo] target(s) in 1.00s
Running /Users/ABeingessner/dev/lists/target/debug/deps/lists-86544f1d97438f1f
running 1 test
test first::test::basics ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
; 0 filtered out
やったー！何もしないテストに合格しました！
では，何かをするテストにしていきましょう．
そのために assert_eq! マクロを利用します．
これは特別なテストの魔法ではありません．
このマクロがすることは，与えられた２つのものを比較し，一致しない場合はプログラムをパニックするということだけです．
パニックを起こすことで，テストハーネスに失敗を示すのです．
mod test {
#[test]
fn basics() {
let mut list = List::new();
// 空のリストが正しい振る舞いをするか
assert_eq!(list.pop(), None);
// リストに要素を詰める
list.push(1);
list.push(2);
list.push(3);
// 削除が機能するか
assert_eq!(list.pop(), Some(3));
assert_eq!(list.pop(), Some(2));
// 何も破損していないことを確かめるために，さらに要素をプッシュする
list.push(4);
list.push(5);
// 削除が機能するか
assert_eq!(list.pop(), Some(5));
assert_eq!(list.pop(), Some(4));
// リストが空になるか
assert_eq!(list.pop(), Some(1));
assert_eq!(list.pop(), None);
}
}
> cargo test
error[E0433]: failed to resolve: use of undeclared type or module `List`
--> src/first.rs:43:24
|
43 | let mut list = List::new();
| ^^^^ use of undeclared type or module `List`
おっと．新しいモジュールを作ったので，リストを使うには明示的にそれを引き込む必要があります．
mod test {
use super::List;
// 他は全部同じ
}
> cargo test
warning: unused import: `super::List`
--> src/first.rs:45:9
|
45 | use super::List;
| ^^^^^^^^^^^
|
= note: #[warn(unused_imports)] on by default
Finished dev [unoptimized + debuginfo] target(s) in 0.43s
Running /Users/ABeingessner/dev/lists/target/debug/deps/lists-86544f1d97438f1f
running 1 test
test first::test::basics ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
; 0 filtered out
やった！
しかし，super::List が使用されていないという警告が出ているのは，どういうことでしょう…？
どう考えてもテストの中で使ってますよね．
…テストの時だけですけどね！ コンパイラを喜ばせるために（そしてユーザに優しくするために）テストを実行する時だけ，テストモジュールをコンパイルするように指示する必要があります．
#[cfg(test)]
mod test {
use super::List;
// 他は全部同じ
}
これでテストは完璧です！
Drop メソッド
スタックを作成し，push を実装し，pop もできるようにし，さらにそれが全て正しく動くことのテストもやりました．
この上さらに drop を実装し，リストの片付けができるようにする必要があるのでしょうか？
技術的には，全く必要ありません．
C++ と同じように，Rust はリソースを使い終わった時に自動的にデストラクタを呼んでクリーンアップを行います．
型は，Drop という トレイト を実装している時にデストラクタを持ちます．
pub trait Drop {
fn drop(&mut self);
}
基本的に「スコープから出たら，さっさと荷物をまとめろ」ということです．
リストの要素の型に Drop が実装されていれば，実際に Drop を実装する必要はありません．
デストラクタを呼び出すだけでいいのです．
List の場合，デストラクタがやることはそのヘッドを drop することですが，その結果 Box<Node> を drop することになる可能性があります．
その全てが自動的に処理されるのですが，一つだけ難点があります．
自動的な処理だとまずいことがあるのです．
単純なリストを考えてみましょう:
list -> A -> B -> C
list がドロップされると，A をドロップしようとし，B をドロップしようとし，C をドロップしようとします．
お分かりいただけたでしょうか…？
処理が再帰的になっているので，スタックを吹っ飛ばしてしまう可能性があるのです．
「これは明らかに末尾再帰的だ．まともな言語であれば，このようなコードでスタックが吹っ飛ぶことはないはずだ」と思われた方もいらっしゃるかもしれません．
これは，実は間違っているのです！
その理由を知るために，コンパイラが List の Drop をどのように行うか，手作業で実装してみましょう:
impl Drop for List {
fn drop(&mut self) {
// 注意: 実際には Rust で明示的に `drop` を呼び出すことはできません．
// コンパイラごっこをしているんです！
self.head.drop(); // 末尾再帰的です - いいね！
}
}
impl Drop for Link {
fn drop(&mut self) {
match *self {
Link::Empty => {} // ヨシ！
Link::More(ref mut boxed_node) => {
boxed_node.drop(); // 末尾再帰的です - いいね！
}
}
}
}
impl Drop for Box<Node> {
fn drop(&mut self) {
self.ptr.drop(); // あっ，末尾再帰的じゃないぞ！
deallocate(self.ptr);
}
}
impl Drop for Node {
fn drop(&mut self) {
self.next.drop();
}
}
Box の中身を deallocate した後にドロップすることはできないので，末尾再帰的な方法でドロップすることは できません．
代わりに，Node を Box から出すのを手動で繰り返す必要があります．
impl Drop for List {
fn drop(&mut self) {
let mut cur_link = mem::replace(&mut self.head, Link::Empty);
// `while let` == "このパターンがマッチしている間は，処理を繰り返す"
while let Link::More(mut boxed_node) = cur_link {
cur_link = mem::replace(&mut boxed_node.next, Link::Empty);
// boxed_node はスコープ外に出るので，ここでドロップされる
// しかし，そのノードの `next` フィールドは `Link::Empty` に設定されています
// したがって，際限なく再帰を繰り返すことはありません
}
}
}
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 1 test
test first::test::basics ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
最高！
Bonus Section For Premature Optimization!
Our implementation of drop is actually very similar to
while let Some(_) = self.pop() { }, which is certainly simpler. How is
it different, and what performance issues could result from it once we start
generalizing our list to store things other than integers?
Click to expand for answer
Pop returns Option<i32>, while our implementation only manipulates Links (Box<Node>). So our implementation only moves around pointers to nodes, while the pop-based one will move around the values we stored in nodes. This could be very expensive if we generalize our list and someone uses it to store instances of VeryBigThingWithADropImpl (VBTWADI). Box is able to run the drop implementation of its contents in-place, so it doesn't suffer from this issue. Since VBTWADI is exactly the kind of thing that actually makes using a linked-list desirable over an array, behaving poorly on this case would be a bit of a disappointment.
If you wish to have the best of both implementations, you could add a new method,
fn pop_node(&mut self) -> Link, from-which pop and drop can both be cleanly derived.
出来上がり
さて，6000 字を費やして，実際に書き上げたコードは以下の通りです:
#![allow(unused)] fn main() { use std::mem; pub struct List { head: Link, } enum Link { Empty, More(Box<Node>), } struct Node { elem: i32, next: Link, } impl List { pub fn new() -> Self { List { head: Link::Empty } } pub fn push(&mut self, elem: i32) { let new_node = Box::new(Node { elem: elem, next: mem::replace(&mut self.head, Link::Empty), }); self.head = Link::More(new_node); } pub fn pop(&mut self) -> Option<i32> { match mem::replace(&mut self.head, Link::Empty) { Link::Empty => None, Link::More(node) => { self.head = node.next; Some(node.elem) } } } } impl Drop for List { fn drop(&mut self) { let mut cur_link = mem::replace(&mut self.head, Link::Empty); while let Link::More(mut boxed_node) = cur_link { cur_link = mem::replace(&mut boxed_node.next, Link::Empty); } } } #[cfg(test)] mod test { use super::List; #[test] fn basics() { let mut list = List::new(); // Check empty list behaves right assert_eq!(list.pop(), None); // Populate list list.push(1); list.push(2); list.push(3); // Check normal removal assert_eq!(list.pop(), Some(3)); assert_eq!(list.pop(), Some(2)); // Push some more just to make sure nothing's corrupted list.push(4); list.push(5); // Check normal removal assert_eq!(list.pop(), Some(5)); assert_eq!(list.pop(), Some(4)); // Check exhaustion assert_eq!(list.pop(), Some(1)); assert_eq!(list.pop(), None); } } }
マジかよ，たった 80 行, それも半分がテストコードだなんて！ 確かにこの最初の章は時間がかかるって言ったけどさ！
An Ok Singly-Linked Stack
In the previous chapter we wrote up a minimum viable singly-linked stack. However there's a few design decisions that make it kind of sucky. Let's make it less sucky. In doing so, we will:
- Deinvent the wheel
- Make our list able to handle any element type
- Add peeking
- Make our list iterable
And in the process we'll learn about
- Advanced Option use
- Generics
- Lifetimes
- Iterators
Let's add a new file called second.rs:
// in lib.rs
pub mod first;
pub mod second;
And copy everything from first.rs into it.
Using Option
Particularly observant readers may have noticed that we actually reinvented a really bad version of Option:
enum Link {
Empty,
More(Box<Node>),
}
Link is just Option<Box<Node>>. Now, it's nice not to have to write
Option<Box<Node>> everywhere, and unlike pop, we're not exposing this
to the outside world, so maybe it's fine. However Option has some really
nice methods that we've been manually implementing ourselves. Let's not
do that, and replace everything with Options. First, we'll do it naively
by just renaming everything to use Some and None:
use std::mem;
pub struct List {
head: Link,
}
// yay type aliases!
type Link = Option<Box<Node>>;
struct Node {
elem: i32,
next: Link,
}
impl List {
pub fn new() -> Self {
List { head: None }
}
pub fn push(&mut self, elem: i32) {
let new_node = Box::new(Node {
elem: elem,
next: mem::replace(&mut self.head, None),
});
self.head = Some(new_node);
}
pub fn pop(&mut self) -> Option<i32> {
match mem::replace(&mut self.head, None) {
None => None,
Some(node) => {
self.head = node.next;
Some(node.elem)
}
}
}
}
impl Drop for List {
fn drop(&mut self) {
let mut cur_link = mem::replace(&mut self.head, None);
while let Some(mut boxed_node) = cur_link {
cur_link = mem::replace(&mut boxed_node.next, None);
}
}
}
This is marginally better, but the big wins will come from Option's methods.
First, mem::replace(&mut option, None) is such an incredibly
common idiom that Option actually just went ahead and made it a method: take.
pub struct List {
head: Link,
}
type Link = Option<Box<Node>>;
struct Node {
elem: i32,
next: Link,
}
impl List {
pub fn new() -> Self {
List { head: None }
}
pub fn push(&mut self, elem: i32) {
let new_node = Box::new(Node {
elem: elem,
next: self.head.take(),
});
self.head = Some(new_node);
}
pub fn pop(&mut self) -> Option<i32> {
match self.head.take() {
None => None,
Some(node) => {
self.head = node.next;
Some(node.elem)
}
}
}
}
impl Drop for List {
fn drop(&mut self) {
let mut cur_link = self.head.take();
while let Some(mut boxed_node) = cur_link {
cur_link = boxed_node.next.take();
}
}
}
Second, match option { None => None, Some(x) => Some(y) } is such an
incredibly common idiom that it was called map. map takes a function to
execute on the x in the Some(x) to produce the y in Some(y). We could
write a proper fn and pass it to map, but we'd much rather write what to
do inline.
The way to do this is with a closure. Closures are anonymous functions with
an extra super-power: they can refer to local variables outside the closure!
This makes them super useful for doing all sorts of conditional logic. The
only place we do a match is in pop, so let's just rewrite that:
pub fn pop(&mut self) -> Option<i32> {
self.head.take().map(|node| {
self.head = node.next;
node.elem
})
}
Ah, much better. Let's make sure we didn't break anything:
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 2 tests
test first::test::basics ... ok
test second::test::basics ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
Great! Let's move on to actually improving the code's behaviour.
Making it all Generic
We've already touched a bit on generics with Option and Box. However so far we've managed to avoid declaring any new type that is actually generic over arbitrary elements.
It turns out that's actually really easy. Let's make all of our types generic right now:
pub struct List<T> {
head: Link<T>,
}
type Link<T> = Option<Box<Node<T>>>;
struct Node<T> {
elem: T,
next: Link<T>,
}
You just make everything a little more pointy, and suddenly your code is generic. Of course, we can't just do this, or else the compiler's going to be Super Mad.
> cargo test
error[E0107]: wrong number of type arguments: expected 1, found 0
--> src/second.rs:14:6
|
14 | impl List {
| ^^^^ expected 1 type argument
error[E0107]: wrong number of type arguments: expected 1, found 0
--> src/second.rs:36:15
|
36 | impl Drop for List {
| ^^^^ expected 1 type argument
The problem is pretty clear: we're talking about this List thing but that's not
real anymore. Like Option and Box, we now always have to talk about
List<Something>.
But what's the Something we use in all these impls? Just like List, we want our
implementations to work with all the T's. So, just like List, let's make our
impls pointy:
impl<T> List<T> {
pub fn new() -> Self {
List { head: None }
}
pub fn push(&mut self, elem: T) {
let new_node = Box::new(Node {
elem: elem,
next: self.head.take(),
});
self.head = Some(new_node);
}
pub fn pop(&mut self) -> Option<T> {
self.head.take().map(|node| {
self.head = node.next;
node.elem
})
}
}
impl<T> Drop for List<T> {
fn drop(&mut self) {
let mut cur_link = self.head.take();
while let Some(mut boxed_node) = cur_link {
cur_link = boxed_node.next.take();
}
}
}
...and that's it!
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 2 tests
test first::test::basics ... ok
test second::test::basics ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
All of our code is now completely generic over arbitrary values of T. Dang,
Rust is easy. I'd like to make a particular shout-out to new which didn't
even change:
pub fn new() -> Self {
List { head: None }
}
Bask in the Glory that is Self, guardian of refactoring and copy-pasta coding.
Also of interest, we don't write List<T> when we construct an instance of
list. That part's inferred for us based on the fact that we're returning it
from a function that expects a List<T>.
Alright, let's move on to totally new behaviour!
Peek
One thing we didn't even bother to implement last time was peeking. Let's go ahead and do that. All we need to do is return a reference to the element in the head of the list, if it exists. Sounds easy, let's try:
pub fn peek(&self) -> Option<&T> {
self.head.map(|node| {
&node.elem
})
}
> cargo build
error[E0515]: cannot return reference to local data `node.elem`
--> src/second.rs:37:13
|
37 | &node.elem
| ^^^^^^^^^^ returns a reference to data owned by the current function
error[E0507]: cannot move out of borrowed content
--> src/second.rs:36:9
|
36 | self.head.map(|node| {
| ^^^^^^^^^ cannot move out of borrowed content
Sigh. What now, Rust?
Map takes self by value, which would move the Option out of the thing it's in.
Previously this was fine because we had just taken it out, but now we actually
want to leave it where it was. The correct way to handle this is with the
as_ref method on Option, which has the following definition:
impl<T> Option<T> {
pub fn as_ref(&self) -> Option<&T>;
}
It demotes the Option<T> to an Option to a reference to its internals. We could
do this ourselves with an explicit match but ugh no. It does mean that we
need to do an extra dereference to cut through the extra indirection, but
thankfully the . operator handles that for us.
pub fn peek(&self) -> Option<&T> {
self.head.as_ref().map(|node| {
&node.elem
})
}
cargo build
Finished dev [unoptimized + debuginfo] target(s) in 0.32s
Nailed it.
We can also make a mutable version of this method using as_mut:
pub fn peek_mut(&mut self) -> Option<&mut T> {
self.head.as_mut().map(|node| {
&mut node.elem
})
}
> cargo build
EZ
Don't forget to test it:
#[test]
fn peek() {
let mut list = List::new();
assert_eq!(list.peek(), None);
assert_eq!(list.peek_mut(), None);
list.push(1); list.push(2); list.push(3);
assert_eq!(list.peek(), Some(&3));
assert_eq!(list.peek_mut(), Some(&mut 3));
}
cargo test
Running target/debug/lists-5c71138492ad4b4a
running 3 tests
test first::test::basics ... ok
test second::test::basics ... ok
test second::test::peek ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured
That's nice, but we didn't really test to see if we could mutate that peek_mut return value, did we? If a reference is mutable but nobody mutates it, have we really tested the mutability? Let's try using map on this Option<&mut T> to put a profound value in:
#[test]
fn peek() {
let mut list = List::new();
assert_eq!(list.peek(), None);
assert_eq!(list.peek_mut(), None);
list.push(1); list.push(2); list.push(3);
assert_eq!(list.peek(), Some(&3));
assert_eq!(list.peek_mut(), Some(&mut 3));
list.peek_mut().map(|&mut value| {
value = 42
});
assert_eq!(list.peek(), Some(&42));
assert_eq!(list.pop(), Some(42));
}
> cargo test
error[E0384]: cannot assign twice to immutable variable `value`
--> src/second.rs:100:13
|
99 | list.peek_mut().map(|&mut value| {
| -----
| |
| first assignment to `value`
| help: make this binding mutable: `mut value`
100 | value = 42
| ^^^^^^^^^^ cannot assign twice to immutable variable ^~~~~
The compiler is complaining that value is immutable, but we pretty clearly wrote &mut value; what gives? It turns out that writing the argument of the closure that way doesn't specify that value is a mutable reference. Instead, it creates a pattern that will be matched against the argument to the closure; |&mut value| means "the argument is a mutable reference, but just copy the value it points to into value, please." If we just use |value|, the type of value will be &mut i32 and we can actually mutate the head:
#[test]
fn peek() {
let mut list = List::new();
assert_eq!(list.peek(), None);
assert_eq!(list.peek_mut(), None);
list.push(1); list.push(2); list.push(3);
assert_eq!(list.peek(), Some(&3));
assert_eq!(list.peek_mut(), Some(&mut 3));
list.peek_mut().map(|value| {
*value = 42
});
assert_eq!(list.peek(), Some(&42));
assert_eq!(list.pop(), Some(42));
}
cargo test
Running target/debug/lists-5c71138492ad4b4a
running 3 tests
test first::test::basics ... ok
test second::test::basics ... ok
test second::test::peek ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured
Much better!
IntoIter
Collections are iterated in Rust using the Iterator trait. It's a bit more
complicated than Drop:
pub trait Iterator {
type Item;
fn next(&mut self) -> Option<Self::Item>;
}
The new kid on the block here is type Item. This is declaring that every
implementation of Iterator has an associated type called Item. In this case,
this is the type that it can spit out when you call next.
The reason Iterator yields Option<Self::Item> is because the interface
coalesces the has_next and get_next concepts. When you have the next value,
you yield
Some(value), and when you don't you yield None. This makes the
API generally more ergonomic and safe to use and implement, while avoiding
redundant checks and logic between has_next and get_next. Nice!
Sadly, Rust has nothing like a yield statement (yet), so we're going to have to
implement the logic ourselves. Also, there's actually 3 different kinds of
iterator each collection should endeavour to implement:
- IntoIter -
T - IterMut -
&mut T - Iter -
&T
We actually already have all the tools to implement
IntoIter using List's interface: just call pop over and over. As such, we'll
just implement IntoIter as a newtype wrapper around List:
// Tuple structs are an alternative form of struct,
// useful for trivial wrappers around other types.
pub struct IntoIter<T>(List<T>);
impl<T> List<T> {
pub fn into_iter(self) -> IntoIter<T> {
IntoIter(self)
}
}
impl<T> Iterator for IntoIter<T> {
type Item = T;
fn next(&mut self) -> Option<Self::Item> {
// access fields of a tuple struct numerically
self.0.pop()
}
}
And let's write a test:
#[test]
fn into_iter() {
let mut list = List::new();
list.push(1); list.push(2); list.push(3);
let mut iter = list.into_iter();
assert_eq!(iter.next(), Some(3));
assert_eq!(iter.next(), Some(2));
assert_eq!(iter.next(), Some(1));
assert_eq!(iter.next(), None);
}
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 4 tests
test first::test::basics ... ok
test second::test::basics ... ok
test second::test::into_iter ... ok
test second::test::peek ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured
Nice!
Iter
Alright, let's try to implement Iter. This time we won't be able to rely on
List giving us all the features we want. We'll need to roll our own. The
basic logic we want is to hold a pointer to the current node we want to yield
next. Because that node may not exist (the list is empty or we're otherwise
done iterating), we want that reference to be an Option. When we yield an
element, we want to proceed to the current node's next node.
Alright, let's try that:
pub struct Iter<T> {
next: Option<&Node<T>>,
}
impl<T> List<T> {
pub fn iter(&self) -> Iter<T> {
Iter { next: self.head.map(|node| &node) }
}
}
impl<T> Iterator for Iter<T> {
type Item = &T;
fn next(&mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.next.map(|node| &node);
&node.elem
})
}
}
> cargo build
error[E0106]: missing lifetime specifier
--> src/second.rs:72:18
|
72 | next: Option<&Node<T>>,
| ^ expected lifetime parameter
error[E0106]: missing lifetime specifier
--> src/second.rs:82:17
|
82 | type Item = &T;
| ^ expected lifetime parameter
Oh god. Lifetimes. I've heard of these things. I hear they're a nightmare.
Let's try something new: see that error[E0106] thing? That's a compiler error
code. We can ask rustc to explain those with, well, --explain:
> rustc --explain E0106
This error indicates that a lifetime is missing from a type. If it is an error
inside a function signature, the problem may be with failing to adhere to the
lifetime elision rules (see below).
Here are some simple examples of where you'll run into this error:
struct Foo { x: &bool } // error
struct Foo<'a> { x: &'a bool } // correct
enum Bar { A(u8), B(&bool), } // error
enum Bar<'a> { A(u8), B(&'a bool), } // correct
type MyStr = &str; // error
type MyStr<'a> = &'a str; //correct
...
That uh... that didn't really clarify much (these docs assume we understand
Rust better than we currently do). But it looks like we should add
those 'a things to our struct? Let's try that.
pub struct Iter<'a, T> {
next: Option<&'a Node<T>>,
}
> cargo build
error[E0106]: missing lifetime specifier
--> src/second.rs:83:22
|
83 | impl<T> Iterator for Iter<T> {
| ^^^^^^^ expected lifetime parameter
error[E0106]: missing lifetime specifier
--> src/second.rs:84:17
|
84 | type Item = &T;
| ^ expected lifetime parameter
error: aborting due to 2 previous errors
Alright I'm starting to see a pattern here... let's just add these little guys to everything we can:
pub struct Iter<'a, T> {
next: Option<&'a Node<T>>,
}
impl<'a, T> List<T> {
pub fn iter(&'a self) -> Iter<'a, T> {
Iter { next: self.head.map(|node| &'a node) }
}
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&'a mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.next.map(|node| &'a node);
&'a node.elem
})
}
}
> cargo build
error: expected `:`, found `node`
--> src/second.rs:77:47
|
77 | Iter { next: self.head.map(|node| &'a node) }
| ---- while parsing this struct ^^^^ expected `:`
error: expected `:`, found `node`
--> src/second.rs:85:50
|
85 | self.next = node.next.map(|node| &'a node);
| ^^^^ expected `:`
error[E0063]: missing field `next` in initializer of `second::Iter<'_, _>`
--> src/second.rs:77:9
|
77 | Iter { next: self.head.map(|node| &'a node) }
| ^^^^ missing `next`
Oh god. We broke Rust.
Maybe we should actually figure out what the heck this 'a lifetime stuff
even means.
Lifetimes can scare off a lot of people because they're a change to something we've known and loved since the dawn of programming. We've actually managed to dodge lifetimes so far, even though they've been tangled throughout our programs this whole time.
Lifetimes are unnecessary in garbage collected languages because the garbage collector ensures that everything magically lives as long as it needs to. Most data in Rust is manually managed, so that data needs another solution. C and C++ give us a clear example what happens if you just let people take pointers to random data on the stack: pervasive unmanageable unsafety. This can be roughly separated into two classes of error:
- Holding a pointer to something that went out of scope
- Holding a pointer to something that got mutated away
Lifetimes solve both of these problems, and 99% of the time, they do this in a totally transparent way.
So what's a lifetime?
Quite simply, a lifetime is the name of a region (~block/scope) of code somewhere in a program. That's it. When a reference is tagged with a lifetime, we're saying that it has to be valid for that entire region. Different things place requirements on how long a reference must and can be valid for. The entire lifetime system is in turn just a constraint-solving system that tries to minimize the region of every reference. If it successfully finds a set of lifetimes that satisfies all the constraints, your program compiles! Otherwise you get an error back saying that something didn't live long enough.
Within a function body you generally can't talk about lifetimes, and wouldn't want to anyway. The compiler has full information and can infer all the constraints to find the minimum lifetimes. However at the type and API-level, the compiler doesn't have all the information. It requires you to tell it about the relationship between different lifetimes so it can figure out what you're doing.
In principle, those lifetimes could also be left out, but then checking all the borrows would be a huge whole-program analysis that would produce mind-bogglingly non-local errors. Rust's system means all borrow checking can be done in each function body independently, and all your errors should be fairly local (or your types have incorrect signatures).
But we've written references in function signatures before, and it was fine! That's because there are certain cases that are so common that Rust will automatically pick the lifetimes for you. This is lifetime elision.
In particular:
// Only one reference in input, so the output must be derived from that input
fn foo(&A) -> &B; // sugar for:
fn foo<'a>(&'a A) -> &'a B;
// Many inputs, assume they're all independent
fn foo(&A, &B, &C); // sugar for:
fn foo<'a, 'b, 'c>(&'a A, &'b B, &'c C);
// Methods, assume all output lifetimes are derived from `self`
fn foo(&self, &B, &C) -> &D; // sugar for:
fn foo<'a, 'b, 'c>(&'a self, &'b B, &'c C) -> &'a D;
So what does fn foo<'a>(&'a A) -> &'a B mean? In practical terms, all it
means is that the input must live at least as long as the output. So if you keep
the output around for a long time, this will expand the region that the input must
be valid for. Once you stop using the output, the compiler will know it's ok for
the input to become invalid too.
With this system set up, Rust can ensure nothing is used after free, and nothing is mutated while outstanding references exist. It just makes sure the constraints all work out!
Alright. So. Iter.
Let's roll back to the no lifetimes state:
pub struct Iter<T> {
next: Option<&Node<T>>,
}
impl<T> List<T> {
pub fn iter(&self) -> Iter<T> {
Iter { next: self.head.map(|node| &node) }
}
}
impl<T> Iterator for Iter<T> {
type Item = &T;
fn next(&mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.next.map(|node| &node);
&node.elem
})
}
}
We need to add lifetimes only in function and type signatures:
// Iter is generic over *some* lifetime, it doesn't care
pub struct Iter<'a, T> {
next: Option<&'a Node<T>>,
}
// No lifetime here, List doesn't have any associated lifetimes
impl<T> List<T> {
// We declare a fresh lifetime here for the *exact* borrow that
// creates the iter. Now &self needs to be valid as long as the
// Iter is around.
pub fn iter<'a>(&'a self) -> Iter<'a, T> {
Iter { next: self.head.map(|node| &node) }
}
}
// We *do* have a lifetime here, because Iter has one that we need to define
impl<'a, T> Iterator for Iter<'a, T> {
// Need it here too, this is a type declaration
type Item = &'a T;
// None of this needs to change, handled by the above.
// Self continues to be incredibly hype and amazing
fn next(&mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.next.map(|node| &node);
&node.elem
})
}
}
Alright, I think we got it this time y'all.
cargo build
error[E0308]: mismatched types
--> src/second.rs:77:22
|
77 | Iter { next: self.head.map(|node| &node) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `second::Node`, found struct `std::boxed::Box`
|
= note: expected type `std::option::Option<&second::Node<T>>`
found type `std::option::Option<&std::boxed::Box<second::Node<T>>>`
error[E0308]: mismatched types
--> src/second.rs:85:25
|
85 | self.next = node.next.map(|node| &node);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `second::Node`, found struct `std::boxed::Box`
|
= note: expected type `std::option::Option<&'a second::Node<T>>`
found type `std::option::Option<&std::boxed::Box<second::Node<T>>>`
(╯°□°)╯︵ ┻━┻
OK. SO. We fixed our lifetime errors but now we're getting some new type errors.
We want to be storing &Node's, but we're getting &Box<Node>s. Ok, that's easy
enough, we just need to dereference the Box before we take our reference:
impl<T> List<T> {
pub fn iter<'a>(&'a self) -> Iter<'a, T> {
Iter { next: self.head.map(|node| &*node) }
}
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.next.map(|node| &*node);
&node.elem
})
}
}
cargo build
Compiling lists v0.1.0 (/Users/ABeingessner/dev/temp/lists)
error[E0515]: cannot return reference to local data `*node`
--> src/second.rs:77:43
|
77 | Iter { next: self.head.map(|node| &*node) }
| ^^^^^^ returns a reference to data owned by the current function
error[E0507]: cannot move out of borrowed content
--> src/second.rs:77:22
|
77 | Iter { next: self.head.map(|node| &*node) }
| ^^^^^^^^^ cannot move out of borrowed content
error[E0515]: cannot return reference to local data `*node`
--> src/second.rs:85:46
|
85 | self.next = node.next.map(|node| &*node);
| ^^^^^^ returns a reference to data owned by the current function
error[E0507]: cannot move out of borrowed content
--> src/second.rs:85:25
|
85 | self.next = node.next.map(|node| &*node);
| ^^^^^^^^^ cannot move out of borrowed content
(ﾉಥ益ಥ）ﾉ ┻━┻
We forgot as_ref, so we're moving the box into map, which means it would
be dropped, which means our references would be dangling:
pub struct Iter<'a, T> {
next: Option<&'a Node<T>>,
}
impl<T> List<T> {
pub fn iter<'a>(&'a self) -> Iter<'a, T> {
Iter { next: self.head.as_ref().map(|node| &*node) }
}
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.next.as_ref().map(|node| &*node);
&node.elem
})
}
}
cargo build
Compiling lists v0.1.0 (/Users/ABeingessner/dev/temp/lists)
error[E0308]: mismatched types
--> src/second.rs:77:22
|
77 | Iter { next: self.head.as_ref().map(|node| &*node) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `second::Node`, found struct `std::boxed::Box`
|
= note: expected type `std::option::Option<&second::Node<T>>`
found type `std::option::Option<&std::boxed::Box<second::Node<T>>>`
error[E0308]: mismatched types
--> src/second.rs:85:25
|
85 | self.next = node.next.as_ref().map(|node| &*node);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `second::Node`, found struct `std::boxed::Box`
|
= note: expected type `std::option::Option<&'a second::Node<T>>`
found type `std::option::Option<&std::boxed::Box<second::Node<T>>>`
😭
as_ref added another layer of indirection we need to remove:
pub struct Iter<'a, T> {
next: Option<&'a Node<T>>,
}
impl<T> List<T> {
pub fn iter<'a>(&'a self) -> Iter<'a, T> {
Iter { next: self.head.as_deref() }
}
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.next.as_deref();
&node.elem
})
}
}
cargo build
🎉 🎉 🎉
The as_deref and as_deref_mut functions are stable as of Rust 1.40. Before that you
would need to do map(|node| &**node) and map(|node| &mut**node).
You may be thinking "wow that &** thing is really janky", and you're not wrong,
but like a fine wine Rust gets better over time and we no longer need to do such.
Normally Rust is very good at doing this kind of conversion implicitly, through
a process called deref coercion, where basically it can insert *'s
throughout your code to make it type-check. It can do this because we have the
borrow checker to ensure we never mess up pointers!
But in this case the closure in conjunction with the fact that we
have an Option<&T> instead of &T is a bit too complicated for it to work
out, so we need to help it by being explicit. Thankfully this is pretty rare, in my experience.
Just for completeness' sake, we could give it a different hint with the turbofish:
self.next = node.next.as_ref().map::<&Node<T>, _>(|node| &node);
See, map is a generic function:
pub fn map<U, F>(self, f: F) -> Option<U>
The turbofish, ::<>, lets us tell the compiler what we think the types of those
generics should be. In this case ::<&Node<T>, _> says "it should return a
&Node<T>, and I don't know/care about that other type".
This in turn lets the compiler know that &node should have deref coercion
applied to it, so we don't need to manually apply all those *'s!
But in this case I don't think it's really an improvement, this was just a thinly veiled excuse to show off deref coercion and the sometimes-useful turbofish. 😅
Let's write a test to be sure we didn't no-op it or anything:
#[test]
fn iter() {
let mut list = List::new();
list.push(1); list.push(2); list.push(3);
let mut iter = list.iter();
assert_eq!(iter.next(), Some(&3));
assert_eq!(iter.next(), Some(&2));
assert_eq!(iter.next(), Some(&1));
}
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 5 tests
test first::test::basics ... ok
test second::test::basics ... ok
test second::test::into_iter ... ok
test second::test::iter ... ok
test second::test::peek ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured
Heck yeah.
Finally, it should be noted that we can actually apply lifetime elision here:
impl<T> List<T> {
pub fn iter<'a>(&'a self) -> Iter<'a, T> {
Iter { next: self.head.as_deref() }
}
}
is equivalent to:
impl<T> List<T> {
pub fn iter(&self) -> Iter<T> {
Iter { next: self.head.as_deref() }
}
}
Yay fewer lifetimes!
Or, if you're not comfortable "hiding" that a struct contains a lifetime,
you can use the Rust 2018 "explicitly elided lifetime" syntax, '_:
impl<T> List<T> {
pub fn iter(&self) -> Iter<'_, T> {
Iter { next: self.head.as_deref() }
}
}
IterMut
I'm gonna be honest, IterMut is WILD. Which in itself seems like a wild thing to say; surely it's identical to Iter!
Semantically, yes, but the nature of shared and mutable references means that Iter is "trivial" while IterMut is Legit Wizard Magic.
The key insight comes from our implementation of Iterator for Iter:
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&mut self) -> Option<Self::Item> { /* stuff */ }
}
Which can be desugared to:
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next<'b>(&'b mut self) -> Option<&'a T> { /* stuff */ }
}
The signature of next establishes no constraint between the lifetime
of the input and the output! Why do we care? It means we can call next
over and over unconditionally!
let mut list = List::new();
list.push(1); list.push(2); list.push(3);
let mut iter = list.iter();
let x = iter.next().unwrap();
let y = iter.next().unwrap();
let z = iter.next().unwrap();
Cool!
This is definitely fine for shared references because the whole point is that you can have tons of them at once. However mutable references can't coexist. The whole point is that they're exclusive.
The end result is that it's notably harder to write IterMut using safe code (and we haven't gotten into what that even means yet...). Surprisingly, IterMut can actually be implemented for many structures completely safely!
We'll start by just taking the Iter code and changing everything to be mutable:
pub struct IterMut<'a, T> {
next: Option<&'a mut Node<T>>,
}
impl<T> List<T> {
pub fn iter_mut(&self) -> IterMut<'_, T> {
IterMut { next: self.head.as_deref_mut() }
}
}
impl<'a, T> Iterator for IterMut<'a, T> {
type Item = &'a mut T;
fn next(&mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.next.as_deref_mut();
&mut node.elem
})
}
}
> cargo build
error[E0596]: cannot borrow `self.head` as mutable, as it is behind a `&` reference
--> src/second.rs:95:25
|
94 | pub fn iter_mut(&self) -> IterMut<'_, T> {
| ----- help: consider changing this to be a mutable reference: `&mut self`
95 | IterMut { next: self.head.as_deref_mut() }
| ^^^^^^^^^ `self` is a `&` reference, so the data it refers to cannot be borrowed as mutable
error[E0507]: cannot move out of borrowed content
--> src/second.rs:103:9
|
103 | self.next.map(|node| {
| ^^^^^^^^^ cannot move out of borrowed content
Ok looks like we've got two different errors here. The first one looks really clear
though, it even tells us how to fix it! You can't upgrade a shared reference to a mutable
one, so iter_mut needs to take &mut self. Just a silly copy-paste error.
pub fn iter_mut(&mut self) -> IterMut<'_, T> {
IterMut { next: self.head.as_deref_mut() }
}
What about the other one?
Oops! I actually accidentally made an error when writing the iter impl in
the previous section, and we were just getting lucky that it worked!
We have just had our first run in with the magic of Copy. When we introduced ownership we said that when you move stuff, you can't use it anymore. For some types, this makes perfect sense. Our good friend Box manages an allocation on the heap for us, and we certainly don't want two pieces of code to think that they need to free its memory.
However for other types this is garbage. Integers have no ownership semantics; they're just meaningless numbers! This is why integers are marked as Copy. Copy types are known to be perfectly copyable by a bitwise copy. As such, they have a super power: when moved, the old value is still usable. As a consequence, you can even move a Copy type out of a reference without replacement!
All numeric primitives in Rust (i32, u64, bool, f32, char, etc...) are Copy. You can also declare any user-defined type to be Copy as well, as long as all its components are Copy.
Critically to why this code was working, shared references are also Copy!
Because & is copy, Option<&> is also Copy. So when we did self.next.map it
was fine because the Option was just copied. Now we can't do that, because
&mut isn't Copy (if you copied an &mut, you'd have two &mut's to the same
location in memory, which is forbidden). Instead, we should properly take
the Option to get it.
fn next(&mut self) -> Option<Self::Item> {
self.next.take().map(|node| {
self.next = node.next.as_deref_mut();
&mut node.elem
})
}
> cargo build
Uh... wow. Holy shit! IterMut Just Works!
Let's test this:
#[test]
fn iter_mut() {
let mut list = List::new();
list.push(1); list.push(2); list.push(3);
let mut iter = list.iter_mut();
assert_eq!(iter.next(), Some(&mut 3));
assert_eq!(iter.next(), Some(&mut 2));
assert_eq!(iter.next(), Some(&mut 1));
}
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 6 tests
test first::test::basics ... ok
test second::test::basics ... ok
test second::test::iter_mut ... ok
test second::test::into_iter ... ok
test second::test::iter ... ok
test second::test::peek ... ok
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured
Yep. It works.
Holy shit.
What.
Ok I mean it actually is supposed to work, but there's usually something stupid that gets in the way! Let's be clear here:
We have just implemented a piece of code that takes a singly-linked list, and returns a mutable reference to every single element in the list at most once. And it's statically verified to do that. And it's totally safe. And we didn't have to do anything wild.
That's kind of a big deal, if you ask me. There are a couple reasons why this works:
- We
taketheOption<&mut>so we have exclusive access to the mutable reference. No need to worry about someone looking at it again. - Rust understands that it's ok to shard a mutable reference into the subfields of the pointed-to struct, because there's no way to "go back up", and they're definitely disjoint.
It turns out that you can apply this basic logic to get a safe IterMut for an array or a tree as well! You can even make the iterator DoubleEnded, so that you can consume the iterator from the front and the back at once! Woah!
Final Code
Alright, that's it for the second list; here's the final code!
#![allow(unused)] fn main() { pub struct List<T> { head: Link<T>, } type Link<T> = Option<Box<Node<T>>>; struct Node<T> { elem: T, next: Link<T>, } impl<T> List<T> { pub fn new() -> Self { List { head: None } } pub fn push(&mut self, elem: T) { let new_node = Box::new(Node { elem: elem, next: self.head.take(), }); self.head = Some(new_node); } pub fn pop(&mut self) -> Option<T> { self.head.take().map(|node| { self.head = node.next; node.elem }) } pub fn peek(&self) -> Option<&T> { self.head.as_ref().map(|node| { &node.elem }) } pub fn peek_mut(&mut self) -> Option<&mut T> { self.head.as_mut().map(|node| { &mut node.elem }) } pub fn into_iter(self) -> IntoIter<T> { IntoIter(self) } pub fn iter(&self) -> Iter<'_, T> { Iter { next: self.head.as_deref() } } pub fn iter_mut(&mut self) -> IterMut<'_, T> { IterMut { next: self.head.as_deref_mut() } } } impl<T> Drop for List<T> { fn drop(&mut self) { let mut cur_link = self.head.take(); while let Some(mut boxed_node) = cur_link { cur_link = boxed_node.next.take(); } } } pub struct IntoIter<T>(List<T>); impl<T> Iterator for IntoIter<T> { type Item = T; fn next(&mut self) -> Option<Self::Item> { // access fields of a tuple struct numerically self.0.pop() } } pub struct Iter<'a, T> { next: Option<&'a Node<T>>, } impl<'a, T> Iterator for Iter<'a, T> { type Item = &'a T; fn next(&mut self) -> Option<Self::Item> { self.next.map(|node| { self.next = node.next.as_deref(); &node.elem }) } } pub struct IterMut<'a, T> { next: Option<&'a mut Node<T>>, } impl<'a, T> Iterator for IterMut<'a, T> { type Item = &'a mut T; fn next(&mut self) -> Option<Self::Item> { self.next.take().map(|node| { self.next = node.next.as_deref_mut(); &mut node.elem }) } } #[cfg(test)] mod test { use super::List; #[test] fn basics() { let mut list = List::new(); // Check empty list behaves right assert_eq!(list.pop(), None); // Populate list list.push(1); list.push(2); list.push(3); // Check normal removal assert_eq!(list.pop(), Some(3)); assert_eq!(list.pop(), Some(2)); // Push some more just to make sure nothing's corrupted list.push(4); list.push(5); // Check normal removal assert_eq!(list.pop(), Some(5)); assert_eq!(list.pop(), Some(4)); // Check exhaustion assert_eq!(list.pop(), Some(1)); assert_eq!(list.pop(), None); } #[test] fn peek() { let mut list = List::new(); assert_eq!(list.peek(), None); assert_eq!(list.peek_mut(), None); list.push(1); list.push(2); list.push(3); assert_eq!(list.peek(), Some(&3)); assert_eq!(list.peek_mut(), Some(&mut 3)); list.peek_mut().map(|value| { *value = 42 }); assert_eq!(list.peek(), Some(&42)); assert_eq!(list.pop(), Some(42)); } #[test] fn into_iter() { let mut list = List::new(); list.push(1); list.push(2); list.push(3); let mut iter = list.into_iter(); assert_eq!(iter.next(), Some(3)); assert_eq!(iter.next(), Some(2)); assert_eq!(iter.next(), Some(1)); assert_eq!(iter.next(), None); } #[test] fn iter() { let mut list = List::new(); list.push(1); list.push(2); list.push(3); let mut iter = list.iter(); assert_eq!(iter.next(), Some(&3)); assert_eq!(iter.next(), Some(&2)); assert_eq!(iter.next(), Some(&1)); } #[test] fn iter_mut() { let mut list = List::new(); list.push(1); list.push(2); list.push(3); let mut iter = list.iter_mut(); assert_eq!(iter.next(), Some(&mut 3)); assert_eq!(iter.next(), Some(&mut 2)); assert_eq!(iter.next(), Some(&mut 1)); } } }
Getting beefier!
A Persistent Singly-Linked Stack
Alright, we've mastered the art of mutable singly-linked stacks.
Let's move from single ownership to shared ownership by writing a persistent immutable singly-linked list. This will be exactly the list that functional programmers have come to know and love. You can get the head or the tail and put someone's head on someone else's tail... and... that's basically it. Immutability is a hell of a drug.
In the process we'll largely just become familiar with Rc and Arc, but this will set us up for the next list which will change the game.
Let's add a new file called third.rs:
// in lib.rs
pub mod first;
pub mod second;
pub mod third;
No copy-pasta this time. This is a clean room operation.
Layout
Alright, back to the drawing board on layout.
The most important thing about a persistent list is that you can manipulate the tails of lists basically for free:
For instance, this isn't an uncommon workload to see with a persistent list:
list1 = A -> B -> C -> D
list2 = tail(list1) = B -> C -> D
list3 = push(list2, X) = X -> B -> C -> D
But at the end we want the memory to look like this:
list1 -> A ---+
|
v
list2 ------> B -> C -> D
^
|
list3 -> X ---+
This just can't work with Boxes, because ownership of B is shared. Who
should free it? If I drop list2, does it free B? With boxes we certainly would
expect so!
Functional languages — and indeed almost every other language — get away with this by using garbage collection. With the magic of garbage collection, B will be freed only after everyone stops looking at it. Hooray!
Rust doesn't have anything like the garbage collectors these languages have. They have tracing GC, which will dig through all the memory that's sitting around at runtime and figure out what's garbage automatically. Instead, all Rust has today is reference counting. Reference counting can be thought of as a very simple GC. For many workloads, it has significantly less throughput than a tracing collector, and it completely falls over if you manage to build cycles. But hey, it's all we've got! Thankfully, for our usecase we'll never run into cycles (feel free to try to prove this to yourself — I sure won't).
So how do we do reference-counted garbage collection? Rc! Rc is just like
Box, but we can duplicate it, and its memory will only be freed when all
the Rc's derived from it are dropped. Unfortunately, this flexibility comes at
a serious cost: we can only take a shared reference to its internals. This means
we can't ever really get data out of one of our lists, nor can we mutate them.
So what's our layout gonna look like? Well, previously we had:
pub struct List<T> {
head: Link<T>,
}
type Link<T> = Option<Box<Node<T>>>;
struct Node<T> {
elem: T,
next: Link<T>,
}
Can we just change Box to Rc?
// in third.rs
pub struct List<T> {
head: Link<T>,
}
type Link<T> = Option<Rc<Node<T>>>;
struct Node<T> {
elem: T,
next: Link<T>,
}
cargo build
error[E0412]: cannot find type `Rc` in this scope
--> src/third.rs:5:23
|
5 | type Link<T> = Option<Rc<Node<T>>>;
| ^^ not found in this scope
help: possible candidate is found in another module, you can import it into scope
|
1 | use std::rc::Rc;
|
Oh dang, sick burn. Unlike everything we used for our mutable lists, Rc is so lame that it's not even implicitly imported into every single Rust program. What a loser.
use std::rc::Rc;
cargo build
warning: field is never used: `head`
--> src/third.rs:4:5
|
4 | head: Link<T>,
| ^^^^^^^^^^^^^
|
= note: #[warn(dead_code)] on by default
warning: field is never used: `elem`
--> src/third.rs:10:5
|
10 | elem: T,
| ^^^^^^^
warning: field is never used: `next`
--> src/third.rs:11:5
|
11 | next: Link<T>,
| ^^^^^^^^^^^^^
Seems legit. Rust continues to be completely trivial to write. I bet we can just find-and-replace Box with Rc and call it a day!
...
No. No we can't.
Basics
We already know a lot of the basics of Rust now, so we can do a lot of the simple stuff again.
For the constructor, we can again just copy-paste:
impl<T> List<T> {
pub fn new() -> Self {
List { head: None }
}
}
push and pop don't really make sense anymore. Instead we can provide
prepend and tail, which provide approximately the same thing.
Let's start with prepending. It takes a list and an element, and returns a
List. Like the mutable list case, we want to make a new node, that has the old
list as its next value. The only novel thing is how to get that next value,
because we're not allowed to mutate anything.
The answer to our prayers is the Clone trait. Clone is implemented by almost every type, and provides a generic way to get "another one like this one" that is logically disjoint, given only a shared reference. It's like a copy constructor in C++, but it's never implicitly invoked.
Rc in particular uses Clone as the way to increment the reference count. So rather than moving a Box to be in the sublist, we just clone the head of the old list. We don't even need to match on the head, because Option exposes a Clone implementation that does exactly the thing we want.
Alright, let's give it a shot:
pub fn prepend(&self, elem: T) -> List<T> {
List { head: Some(Rc::new(Node {
elem: elem,
next: self.head.clone(),
}))}
}
> cargo build
warning: field is never used: `elem`
--> src/third.rs:10:5
|
10 | elem: T,
| ^^^^^^^
|
= note: #[warn(dead_code)] on by default
warning: field is never used: `next`
--> src/third.rs:11:5
|
11 | next: Link<T>,
| ^^^^^^^^^^^^^
Wow, Rust is really hard-nosed about actually using fields. It can tell no consumer can ever actually observe the use of these fields! Still, we seem good so far.
tail is the logical inverse of this operation. It takes a list and returns the
whole list with the first element removed. All that is is cloning the second
element in the list (if it exists). Let's try this:
pub fn tail(&self) -> List<T> {
List { head: self.head.as_ref().map(|node| node.next.clone()) }
}
cargo build
error[E0308]: mismatched types
--> src/third.rs:27:22
|
27 | List { head: self.head.as_ref().map(|node| node.next.clone()) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `std::rc::Rc`, found enum `std::option::Option`
|
= note: expected type `std::option::Option<std::rc::Rc<_>>`
found type `std::option::Option<std::option::Option<std::rc::Rc<_>>>`
Hrm, we messed up. map expects us to return a Y, but here we're returning an
Option<Y>. Thankfully, this is another common Option pattern, and we can just
use and_then to let us return an Option.
pub fn tail(&self) -> List<T> {
List { head: self.head.as_ref().and_then(|node| node.next.clone()) }
}
> cargo build
Great.
Now that we have tail, we should probably provide head, which returns a
reference to the first element. That's just peek from the mutable list:
pub fn head(&self) -> Option<&T> {
self.head.as_ref().map(|node| &node.elem)
}
> cargo build
Nice.
That's enough functionality that we can test it:
#[cfg(test)]
mod test {
use super::List;
#[test]
fn basics() {
let list = List::new();
assert_eq!(list.head(), None);
let list = list.prepend(1).prepend(2).prepend(3);
assert_eq!(list.head(), Some(&3));
let list = list.tail();
assert_eq!(list.head(), Some(&2));
let list = list.tail();
assert_eq!(list.head(), Some(&1));
let list = list.tail();
assert_eq!(list.head(), None);
// Make sure empty tail works
let list = list.tail();
assert_eq!(list.head(), None);
}
}
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 5 tests
test first::test::basics ... ok
test second::test::into_iter ... ok
test second::test::basics ... ok
test second::test::iter ... ok
test third::test::basics ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured
Perfect!
Iter is also identical to how it was for our mutable list:
pub struct Iter<'a, T> {
next: Option<&'a Node<T>>,
}
impl<T> List<T> {
pub fn iter(&self) -> Iter<'_, T> {
Iter { next: self.head.as_deref() }
}
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.next.as_deref();
&node.elem
})
}
}
#[test]
fn iter() {
let list = List::new().prepend(1).prepend(2).prepend(3);
let mut iter = list.iter();
assert_eq!(iter.next(), Some(&3));
assert_eq!(iter.next(), Some(&2));
assert_eq!(iter.next(), Some(&1));
}
cargo test
Running target/debug/lists-5c71138492ad4b4a
running 7 tests
test first::test::basics ... ok
test second::test::basics ... ok
test second::test::iter ... ok
test second::test::into_iter ... ok
test second::test::peek ... ok
test third::test::basics ... ok
test third::test::iter ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured
Who ever said dynamic typing was easier?
(chumps did)
Note that we can't implement IntoIter or IterMut for this type. We only have shared access to elements.
Drop
Like the mutable lists, we have a recursive destructor problem. Admittedly, this isn't as bad of a problem for the immutable list: if we ever hit another node that's the head of another list somewhere, we won't recursively drop it. However it's still a thing we should care about, and how to deal with isn't as clear. Here's how we solved it before:
impl<T> Drop for List<T> {
fn drop(&mut self) {
let mut cur_link = self.head.take();
while let Some(mut boxed_node) = cur_link {
cur_link = boxed_node.next.take();
}
}
}
The problem is the body of the loop:
cur_link = boxed_node.next.take();
This is mutating the Node inside the Box, but we can't do that with Rc; it only gives us shared access, because any number of other Rc's could be pointing at it.
But if we know that we're the last list that knows about this node, it would actually be fine to move the Node out of the Rc. Then we could also know when to stop: whenever we can't hoist out the Node.
And look at that, Rc has a method that does exactly this: try_unwrap:
impl<T> Drop for List<T> {
fn drop(&mut self) {
let mut head = self.head.take();
while let Some(node) = head {
if let Ok(mut node) = Rc::try_unwrap(node) {
head = node.next.take();
} else {
break;
}
}
}
}
cargo test
Compiling lists v0.1.0 (/Users/ABeingessner/dev/too-many-lists/lists)
Finished dev [unoptimized + debuginfo] target(s) in 1.10s
Running /Users/ABeingessner/dev/too-many-lists/lists/target/debug/deps/lists-86544f1d97438f1f
running 8 tests
test first::test::basics ... ok
test second::test::basics ... ok
test second::test::into_iter ... ok
test second::test::iter ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test third::test::basics ... ok
test third::test::iter ... ok
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Great! Nice.
Arc
One reason to use an immutable linked list is to share data across threads. After all, shared mutable state is the root of all evil, and one way to solve that is to kill the mutable part forever.
Except our list isn't thread-safe at all. In order to be thread-safe, we need to fiddle with reference counts atomically. Otherwise, two threads could try to increment the reference count, and only one would happen. Then the list could get freed too soon!
In order to get thread safety, we have to use Arc. Arc is completely identical
to Rc except for the fact that reference counts are modified atomically. This
has a bit of overhead if you don't need it, so Rust exposes both.
All we need to do to make our list is replace every reference to Rc with
std::sync::Arc. That's it. We're thread safe. Done!
But this raises an interesting question: how do we know if a type is thread-safe or not? Can we accidentally mess up?
No! You can't mess up thread-safety in Rust!
The reason this is the case is because Rust models thread-safety in a
first-class way with two traits: Send and Sync.
A type is Send if it's safe to move to another thread. A type is Sync if
it's safe to share between multiple threads. That is, if T is Sync, &T is
Send. Safe in this case means it's impossible to cause data races, (not to
be mistaken with the more general issue of race conditions).
These are marker traits, which is a fancy way of saying they're traits that provide absolutely no interface. You either are Send, or you aren't. It's just a property other APIs can require. If you aren't appropriately Send, then it's statically impossible to be sent to a different thread! Sweet!
Send and Sync are also automatically derived traits based on whether you are totally composed of Send and Sync types. It's similar to how you can only implement Copy if you're only made of Copy types, but then we just go ahead and implement it automatically if you are.
Almost every type is Send and Sync. Most types are Send because they totally own their data. Most types are Sync because the only way to share data across threads is to put them behind a shared reference, which makes them immutable!
However there are special types that violate these properties: those that have interior mutability. So far we've only really interacted with inherited mutability (AKA external mutability): the mutability of a value is inherited from the mutability of its container. That is, you can't just randomly mutate some field of a non-mutable value because you feel like it.
Interior mutability types violate this: they let you mutate through a shared reference. There are two major classes of interior mutability: cells, which only work in a single-threaded context; and locks, which work in a multi-threaded context. For obvious reasons, cells are cheaper when you can use them. There's also atomics, which are primitives that act like a lock.
So what does all of this have to do with Rc and Arc? Well, they both use interior mutability for their reference count. Worse, this reference count is shared between every instance! Rc just uses a cell, which means it's not thread safe. Arc uses an atomic, which means it is thread safe. Of course, you can't magically make a type thread safe by putting it in Arc. Arc can only derive thread-safety like any other type.
I really really really don't want to get into the finer details of atomic memory models or non-derived Send implementations. Needless to say, as you get deeper into Rust's thread-safety story, stuff gets more complicated. As a high-level consumer, it all just works and you don't really need to think about it.
Final Code
That's all I really have to say on the immutable stack. We're getting pretty good at implementing lists now!
#![allow(unused)] fn main() { use std::rc::Rc; pub struct List<T> { head: Link<T>, } type Link<T> = Option<Rc<Node<T>>>; struct Node<T> { elem: T, next: Link<T>, } impl<T> List<T> { pub fn new() -> Self { List { head: None } } pub fn prepend(&self, elem: T) -> List<T> { List { head: Some(Rc::new(Node { elem: elem, next: self.head.clone(), }))} } pub fn tail(&self) -> List<T> { List { head: self.head.as_ref().and_then(|node| node.next.clone()) } } pub fn head(&self) -> Option<&T> { self.head.as_ref().map(|node| &node.elem) } pub fn iter(&self) -> Iter<'_, T> { Iter { next: self.head.as_deref() } } } impl<T> Drop for List<T> { fn drop(&mut self) { let mut head = self.head.take(); while let Some(node) = head { if let Ok(mut node) = Rc::try_unwrap(node) { head = node.next.take(); } else { break; } } } } pub struct Iter<'a, T> { next: Option<&'a Node<T>>, } impl<'a, T> Iterator for Iter<'a, T> { type Item = &'a T; fn next(&mut self) -> Option<Self::Item> { self.next.map(|node| { self.next = node.next.as_deref(); &node.elem }) } } #[cfg(test)] mod test { use super::List; #[test] fn basics() { let list = List::new(); assert_eq!(list.head(), None); let list = list.prepend(1).prepend(2).prepend(3); assert_eq!(list.head(), Some(&3)); let list = list.tail(); assert_eq!(list.head(), Some(&2)); let list = list.tail(); assert_eq!(list.head(), Some(&1)); let list = list.tail(); assert_eq!(list.head(), None); // Make sure empty tail works let list = list.tail(); assert_eq!(list.head(), None); } #[test] fn iter() { let list = List::new().prepend(1).prepend(2).prepend(3); let mut iter = list.iter(); assert_eq!(iter.next(), Some(&3)); assert_eq!(iter.next(), Some(&2)); assert_eq!(iter.next(), Some(&1)); } } }
A Bad but Safe Doubly-Linked Deque
Now that we've seen Rc and heard about interior mutability, this gives an interesting thought... maybe we can mutate through an Rc. And if that's the case, maybe we can implement a doubly-linked list totally safely!
In the process we'll become familiar with interior mutability, and probably learn the hard way that safe doesn't mean correct. Doubly-linked lists are hard, and I always make a mistake somewhere.
Let's add a new file called fourth.rs:
// in lib.rs
pub mod first;
pub mod second;
pub mod third;
pub mod fourth;
This will be another clean-room operation, though as usual we'll probably find some logic that applies verbatim again.
Disclaimer: this chapter is basically a demonstration that this is a very bad idea.
Layout
The key to our design is the RefCell type. The heart of
RefCell is a pair of methods:
fn borrow(&self) -> Ref<'_, T>;
fn borrow_mut(&self) -> RefMut<'_, T>;
The rules for borrow and borrow_mut are exactly those of & and &mut:
you can call borrow as many times as you want, but borrow_mut requires
exclusivity.
Rather than enforcing this statically, RefCell enforces them at runtime.
If you break the rules, RefCell will just panic and crash the program.
Why does it return these Ref and RefMut things? Well, they basically behave
like Rcs but for borrowing. They also keep the RefCell borrowed until they go out
of scope. We'll get to that later.
Now with Rc and RefCell we can become... an incredibly verbose pervasively mutable garbage collected language that can't collect cycles! Y-yaaaaay...
Alright, we want to be doubly-linked. This means each node has a pointer to the previous and next node. Also, the list itself has a pointer to the first and last node. This gives us fast insertion and removal on both ends of the list.
So we probably want something like:
use std::rc::Rc;
use std::cell::RefCell;
pub struct List<T> {
head: Link<T>,
tail: Link<T>,
}
type Link<T> = Option<Rc<RefCell<Node<T>>>>;
struct Node<T> {
elem: T,
next: Link<T>,
prev: Link<T>,
}
> cargo build
warning: field is never used: `head`
--> src/fourth.rs:5:5
|
5 | head: Link<T>,
| ^^^^^^^^^^^^^
|
= note: #[warn(dead_code)] on by default
warning: field is never used: `tail`
--> src/fourth.rs:6:5
|
6 | tail: Link<T>,
| ^^^^^^^^^^^^^
warning: field is never used: `elem`
--> src/fourth.rs:12:5
|
12 | elem: T,
| ^^^^^^^
warning: field is never used: `next`
--> src/fourth.rs:13:5
|
13 | next: Link<T>,
| ^^^^^^^^^^^^^
warning: field is never used: `prev`
--> src/fourth.rs:14:5
|
14 | prev: Link<T>,
| ^^^^^^^^^^^^^
Hey, it built! Lots of dead code warnings, but it built! Let's try to use it.
Building Up
Alright, we'll start with building the list. That's pretty straight-forward
with this new system. new is still trivial, just None out all the fields.
Also because it's getting a bit unwieldy, let's break out a Node constructor
too:
impl<T> Node<T> {
fn new(elem: T) -> Rc<RefCell<Self>> {
Rc::new(RefCell::new(Node {
elem: elem,
prev: None,
next: None,
}))
}
}
impl<T> List<T> {
pub fn new() -> Self {
List { head: None, tail: None }
}
}
> cargo build
**A BUNCH OF DEAD CODE WARNINGS BUT IT BUILT**
Yay!
Now let's try to write pushing onto the front of the list. Because doubly-linked lists are significantly more complicated, we're going to need to do a fair bit more work. Where singly-linked list operations could be reduced to an easy one-liner, doubly-linked list ops are fairly complicated.
In particular we now need to specially handle some boundary cases around
empty lists. Most operations will only touch the head or tail pointer.
However when transitioning to or from the empty list, we need to edit
both at once.
An easy way for us to validate if our methods make sense is if we maintain the following invariant: each node should have exactly two pointers to it. Each node in the middle of the list is pointed at by its predecessor and successor, while the nodes on the ends are pointed to by the list itself.
Let's take a crack at it:
pub fn push_front(&mut self, elem: T) {
// new node needs +2 links, everything else should be +0
let new_head = Node::new(elem);
match self.head.take() {
Some(old_head) => {
// non-empty list, need to connect the old_head
old_head.prev = Some(new_head.clone()); // +1 new_head
new_head.next = Some(old_head); // +1 old_head
self.head = Some(new_head); // +1 new_head, -1 old_head
// total: +2 new_head, +0 old_head -- OK!
}
None => {
// empty list, need to set the tail
self.tail = Some(new_head.clone()); // +1 new_head
self.head = Some(new_head); // +1 new_head
// total: +2 new_head -- OK!
}
}
}
cargo build
error[E0609]: no field `prev` on type `std::rc::Rc<std::cell::RefCell<fourth::Node<T>>>`
--> src/fourth.rs:39:26
|
39 | old_head.prev = Some(new_head.clone()); // +1 new_head
| ^^^^ unknown field
error[E0609]: no field `next` on type `std::rc::Rc<std::cell::RefCell<fourth::Node<T>>>`
--> src/fourth.rs:40:26
|
40 | new_head.next = Some(old_head); // +1 old_head
| ^^^^ unknown field
Alright. Compiler error. Good start. Good start.
Why can't we access the prev and next fields on our nodes? It worked before
when we just had an Rc<Node>. Seems like the RefCell is getting in the way.
We should probably check the docs.
Google's "rust refcell"
A mutable memory location with dynamically checked borrow rules
See the module-level documentation for more.
clicks link
Shareable mutable containers.
Values of the
Cell<T>andRefCell<T>types may be mutated through shared references (i.e. the common&Ttype), whereas most Rust types can only be mutated through unique (&mut T) references. We say thatCell<T>andRefCell<T>provide 'interior mutability', in contrast with typical Rust types that exhibit 'inherited mutability'.Cell types come in two flavors:
Cell<T>andRefCell<T>.Cell<T>providesgetandsetmethods that change the interior value with a single method call.Cell<T>though is only compatible with types that implementCopy. For other types, one must use theRefCell<T>type, acquiring a write lock before mutating.
RefCell<T>uses Rust's lifetimes to implement 'dynamic borrowing', a process whereby one can claim temporary, exclusive, mutable access to the inner value. Borrows forRefCell<T>s are tracked 'at runtime', unlike Rust's native reference types which are entirely tracked statically, at compile time. BecauseRefCell<T>borrows are dynamic it is possible to attempt to borrow a value that is already mutably borrowed; when this happens it results in thread panic.When to choose interior mutability
The more common inherited mutability, where one must have unique access to mutate a value, is one of the key language elements that enables Rust to reason strongly about pointer aliasing, statically preventing crash bugs. Because of that, inherited mutability is preferred, and interior mutability is something of a last resort. Since cell types enable mutation where it would otherwise be disallowed though, there are occasions when interior mutability might be appropriate, or even must be used, e.g.
- Introducing inherited mutability roots to shared types.
- Implementation details of logically-immutable methods.
- Mutating implementations of
Clone.Introducing inherited mutability roots to shared types
Shared smart pointer types, including
Rc<T>andArc<T>, provide containers that can be cloned and shared between multiple parties. Because the contained values may be multiply-aliased, they can only be borrowed as shared references, not mutable references. Without cells it would be impossible to mutate data inside of shared boxes at all!It's very common then to put a
RefCell<T>inside shared pointer types to reintroduce mutability:use std::collections::HashMap; use std::cell::RefCell; use std::rc::Rc; fn main() { let shared_map: Rc<RefCell<_>> = Rc::new(RefCell::new(HashMap::new())); shared_map.borrow_mut().insert("africa", 92388); shared_map.borrow_mut().insert("kyoto", 11837); shared_map.borrow_mut().insert("piccadilly", 11826); shared_map.borrow_mut().insert("marbles", 38); }Note that this example uses
Rc<T>and notArc<T>.RefCell<T>s are for single-threaded scenarios. Consider usingMutex<T>if you need shared mutability in a multi-threaded situation.
Hey, Rust's docs continue to be incredibly awesome.
The meaty bit we care about is this line:
shared_map.borrow_mut().insert("africa", 92388);
In particular, the borrow_mut thing. Seems we need to explicitly borrow a
RefCell. The . operator's not going to do it for us. Weird. Let's try:
pub fn push_front(&mut self, elem: T) {
let new_head = Node::new(elem);
match self.head.take() {
Some(old_head) => {
old_head.borrow_mut().prev = Some(new_head.clone());
new_head.borrow_mut().next = Some(old_head);
self.head = Some(new_head);
}
None => {
self.tail = Some(new_head.clone());
self.head = Some(new_head);
}
}
}
> cargo build
warning: field is never used: `elem`
--> src/fourth.rs:12:5
|
12 | elem: T,
| ^^^^^^^
|
= note: #[warn(dead_code)] on by default
Hey, it built! Docs win again.
Breaking Down
pop_front should be the same basic logic as push_front, but backwards. Let's
try:
pub fn pop_front(&mut self) -> Option<T> {
// need to take the old head, ensuring it's -2
self.head.take().map(|old_head| { // -1 old
match old_head.borrow_mut().next.take() {
Some(new_head) => { // -1 new
// not emptying list
new_head.borrow_mut().prev.take(); // -1 old
self.head = Some(new_head); // +1 new
// total: -2 old, +0 new
}
None => {
// emptying list
self.tail.take(); // -1 old
// total: -2 old, (no new)
}
}
old_head.elem
})
}
> cargo build
error[E0609]: no field `elem` on type `std::rc::Rc<std::cell::RefCell<fourth::Node<T>>>`
--> src/fourth.rs:64:22
|
64 | old_head.elem
| ^^^^ unknown field
ACK. RefCells. Gotta borrow_mut again I guess...
pub fn pop_front(&mut self) -> Option<T> {
self.head.take().map(|old_head| {
match old_head.borrow_mut().next.take() {
Some(new_head) => {
new_head.borrow_mut().prev.take();
self.head = Some(new_head);
}
None => {
self.tail.take();
}
}
old_head.borrow_mut().elem
})
}
cargo build
error[E0507]: cannot move out of borrowed content
--> src/fourth.rs:64:13
|
64 | old_head.borrow_mut().elem
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot move out of borrowed content
sigh
cannot move out of borrowed content
Hrm... It seems that Box was really spoiling us. borrow_mut only gets us
an &mut Node<T>, but we can't move out of that!
We need something that takes a RefCell<T> and gives us a T. Let's check
the docs for something like that:
fn into_inner(self) -> TConsumes the RefCell, returning the wrapped value.
That looks promising!
old_head.into_inner().elem
> cargo build
error[E0507]: cannot move out of an `Rc`
--> src/fourth.rs:64:13
|
64 | old_head.into_inner().elem
| ^^^^^^^^ cannot move out of an `Rc`
Ah dang. into_inner wants to move out the RefCell, but we can't, because it's
in an Rc. As we saw in the previous chapter, Rc<T> only lets us get shared references
into its internals. That makes sense, because that's the whole point of
reference counted pointers: they're shared!
This was a problem for us when we wanted to implement Drop for our reference
counted list, and the solution is the same: Rc::try_unwrap, which moves out
the contents of an Rc if its refcount is 1.
Rc::try_unwrap(old_head).unwrap().into_inner().elem
Rc::try_unwrap returns a Result<T, Rc<T>>. Results are basically a
generalized Option, where the None case has data associated with it. In
this case, the Rc you tried to unwrap. Since we don't care about the case
where it fails (if we wrote our program correctly, it has to succeed), we
just call unwrap on it.
Anyway, let's see what compiler error we get next (let's face it, there's going to be one).
> cargo build
error[E0599]: no method named `unwrap` found for type `std::result::Result<std::cell::RefCell<fourth::Node<T>>, std::rc::Rc<std::cell::RefCell<fourth::Node<T>>>>` in the current scope
--> src/fourth.rs:64:38
|
64 | Rc::try_unwrap(old_head).unwrap().into_inner().elem
| ^^^^^^
|
= note: the method `unwrap` exists but the following trait bounds were not satisfied:
`std::rc::Rc<std::cell::RefCell<fourth::Node<T>>> : std::fmt::Debug`
UGH. unwrap on Result requires that you can debug-print the error case.
RefCell<T> only implements Debug if T does. Node doesn't implement Debug.
Rather than doing that, let's just work around it by converting the Result to
an Option with ok:
Rc::try_unwrap(old_head).ok().unwrap().into_inner().elem
PLEASE.
cargo build
YES.
phew
We did it.
We implemented push and pop.
Let's test by stealing the old stack basic test (because that's all that
we've implemented so far):
#[cfg(test)]
mod test {
use super::List;
#[test]
fn basics() {
let mut list = List::new();
// Check empty list behaves right
assert_eq!(list.pop_front(), None);
// Populate list
list.push_front(1);
list.push_front(2);
list.push_front(3);
// Check normal removal
assert_eq!(list.pop_front(), Some(3));
assert_eq!(list.pop_front(), Some(2));
// Push some more just to make sure nothing's corrupted
list.push_front(4);
list.push_front(5);
// Check normal removal
assert_eq!(list.pop_front(), Some(5));
assert_eq!(list.pop_front(), Some(4));
// Check exhaustion
assert_eq!(list.pop_front(), Some(1));
assert_eq!(list.pop_front(), None);
}
}
cargo test
Running target/debug/lists-5c71138492ad4b4a
running 9 tests
test first::test::basics ... ok
test fourth::test::basics ... ok
test second::test::iter_mut ... ok
test second::test::basics ... ok
test fifth::test::iter_mut ... ok
test third::test::basics ... ok
test second::test::iter ... ok
test third::test::iter ... ok
test second::test::into_iter ... ok
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured
Nailed it.
Now that we can properly remove things from the list, we can implement Drop. Drop is a little more conceptually interesting this time around. Where previously we bothered to implement Drop for our stacks just to avoid unbounded recursion, now we need to implement Drop to get anything to happen at all.
Rc can't deal with cycles. If there's a cycle, everything will keep everything
else alive. A doubly-linked list, as it turns out, is just a big chain of tiny
cycles! So when we drop our list, the two end nodes will have their refcounts
decremented down to 1... and then nothing else will happen. Well, if our list
contains exactly one node we're good to go. But ideally a list should work right
if it contains multiple elements. Maybe that's just me.
As we saw, removing elements was a bit painful. So the easiest thing for us to
do is just pop until we get None:
impl<T> Drop for List<T> {
fn drop(&mut self) {
while self.pop_front().is_some() {}
}
}
cargo build
(We actually could have done this with our mutable stacks, but shortcuts are for people who understand things!)
We could look at implementing the _back versions of push and pop, but
they're just copy-paste jobs which we'll defer to later in the chapter. For now
let's look at more interesting things!
Peeking
Alright, we made it through push and pop. I'm not gonna lie, it got a
bit emotional there. Compile-time correctness is a hell of a drug.
Let's cool off by doing something simple: let's just implement peek_front.
That was always really easy before. Gotta still be easy, right?
Right?
In fact, I think I can just copy-paste it!
pub fn peek_front(&self) -> Option<&T> {
self.head.as_ref().map(|node| {
&node.elem
})
}
Wait. Not this time.
pub fn peek_front(&self) -> Option<&T> {
self.head.as_ref().map(|node| {
// BORROW!!!!
&node.borrow().elem
})
}
HAH.
cargo build
error[E0515]: cannot return value referencing temporary value
--> src/fourth.rs:66:13
|
66 | &node.borrow().elem
| ^ ----------^^^^^
| | |
| | temporary value created here
| |
| returns a value referencing data owned by the current function
Ok I'm just burning my computer.
This is exactly the same logic as our singly-linked stack. Why are things different. WHY.
The answer is really the whole moral of this chapter: RefCells make everything sadness. Up until now, RefCells have just been a nuisance. Now they're going to become a nightmare.
So what's going on? To understand that, we need to go back to the definition of
borrow:
fn borrow<'a>(&'a self) -> Ref<'a, T>
fn borrow_mut<'a>(&'a self) -> RefMut<'a, T>
In the layout section we said:
Rather than enforcing this statically, RefCell enforces them at runtime. If you break the rules, RefCell will just panic and crash the program. Why does it return these Ref and RefMut things? Well, they basically behave like
Rcs but for borrowing. Also they keep the RefCell borrowed until they go out of scope. We'll get to that later.
It's later.
Ref and RefMut implement Deref and DerefMut respectively. So for most
intents and purposes they behave exactly like &T and &mut T. However,
because of how those traits work, the reference that's returned is connected
to the lifetime of the Ref, and not the actual RefCell. This means that the Ref
has to be sitting around as long as we keep the reference around.
This is in fact necessary for correctness. When a Ref gets dropped, it tells the RefCell that it's not borrowed anymore. So if we did manage to hold onto our reference longer than the Ref existed, we could get a RefMut while a reference was kicking around and totally break Rust's type system in half.
So where does that leave us? We only want to return a reference, but we need
to keep this Ref thing around. But as soon as we return the reference from
peek, the function is over and the Ref goes out of scope.
😖
As far as I know, we're actually totally dead in the water here. You can't totally encapsulate the use of RefCells like that.
But... what if we just give up on totally hiding our implementation details? What if we returns Refs?
pub fn peek_front(&self) -> Option<Ref<T>> {
self.head.as_ref().map(|node| {
node.borrow()
})
}
> cargo build
error[E0412]: cannot find type `Ref` in this scope
--> src/fourth.rs:63:40
|
63 | pub fn peek_front(&self) -> Option<Ref<T>> {
| ^^^ not found in this scope
help: possible candidates are found in other modules, you can import them into scope
|
1 | use core::cell::Ref;
|
1 | use std::cell::Ref;
|
Blurp. Gotta import some stuff.
use std::cell::{Ref, RefCell};
> cargo build
error[E0308]: mismatched types
--> src/fourth.rs:64:9
|
64 | / self.head.as_ref().map(|node| {
65 | | node.borrow()
66 | | })
| |__________^ expected type parameter, found struct `fourth::Node`
|
= note: expected type `std::option::Option<std::cell::Ref<'_, T>>`
found type `std::option::Option<std::cell::Ref<'_, fourth::Node<T>>>`
Hmm... that's right. We have a Ref<Node<T>>, but we want a Ref<T>. We could
abandon all hope of encapsulation and just return that. We could also make
things even more complicated and wrap Ref<Node<T>> in a new type to only
expose access to an &T.
Both of those options are kinda lame.
Instead, we're going to go deeper down. Let's have some fun. Our source of fun is this beast:
map<U, F>(orig: Ref<'b, T>, f: F) -> Ref<'b, U>
where F: FnOnce(&T) -> &U,
U: ?Sized
Make a new Ref for a component of the borrowed data.
Yes: just like you can map over an Option, you can map over a Ref.
I'm sure someone somewhere is really excited because monads or whatever but I don't care about any of that. Also I don't think it's a proper monad since there's no None-like case, but I digress.
It's cool and that's all that matters to me. I need this.
pub fn peek_front(&self) -> Option<Ref<T>> {
self.head.as_ref().map(|node| {
Ref::map(node.borrow(), |node| &node.elem)
})
}
> cargo build
Awww yissss
Let's make sure this is working by munging up the test from our stack. We need to do some munging to deal with the fact that Refs don't implement comparisons.
#[test]
fn peek() {
let mut list = List::new();
assert!(list.peek_front().is_none());
list.push_front(1); list.push_front(2); list.push_front(3);
assert_eq!(&*list.peek_front().unwrap(), &3);
}
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 10 tests
test first::test::basics ... ok
test fourth::test::basics ... ok
test second::test::basics ... ok
test fourth::test::peek ... ok
test second::test::iter_mut ... ok
test second::test::into_iter ... ok
test third::test::basics ... ok
test second::test::peek ... ok
test second::test::iter ... ok
test third::test::iter ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured
Great!
Symmetric Junk
Alright let's get all that combinatoric symmetry over with.
All we have to do is some basic text replacement:
tail <-> head
next <-> prev
front -> back
Oh, also we need to add _mut variants for peeking.
use std::cell::{Ref, RefCell, RefMut};
//..
pub fn push_back(&mut self, elem: T) {
let new_tail = Node::new(elem);
match self.tail.take() {
Some(old_tail) => {
old_tail.borrow_mut().next = Some(new_tail.clone());
new_tail.borrow_mut().prev = Some(old_tail);
self.tail = Some(new_tail);
}
None => {
self.head = Some(new_tail.clone());
self.tail = Some(new_tail);
}
}
}
pub fn pop_back(&mut self) -> Option<T> {
self.tail.take().map(|old_tail| {
match old_tail.borrow_mut().prev.take() {
Some(new_tail) => {
new_tail.borrow_mut().next.take();
self.tail = Some(new_tail);
}
None => {
self.head.take();
}
}
Rc::try_unwrap(old_tail).ok().unwrap().into_inner().elem
})
}
pub fn peek_back(&self) -> Option<Ref<T>> {
self.tail.as_ref().map(|node| {
Ref::map(node.borrow(), |node| &node.elem)
})
}
pub fn peek_back_mut(&mut self) -> Option<RefMut<T>> {
self.tail.as_ref().map(|node| {
RefMut::map(node.borrow_mut(), |node| &mut node.elem)
})
}
pub fn peek_front_mut(&mut self) -> Option<RefMut<T>> {
self.head.as_ref().map(|node| {
RefMut::map(node.borrow_mut(), |node| &mut node.elem)
})
}
And massively flesh out our tests:
#[test]
fn basics() {
let mut list = List::new();
// Check empty list behaves right
assert_eq!(list.pop_front(), None);
// Populate list
list.push_front(1);
list.push_front(2);
list.push_front(3);
// Check normal removal
assert_eq!(list.pop_front(), Some(3));
assert_eq!(list.pop_front(), Some(2));
// Push some more just to make sure nothing's corrupted
list.push_front(4);
list.push_front(5);
// Check normal removal
assert_eq!(list.pop_front(), Some(5));
assert_eq!(list.pop_front(), Some(4));
// Check exhaustion
assert_eq!(list.pop_front(), Some(1));
assert_eq!(list.pop_front(), None);
// ---- back -----
// Check empty list behaves right
assert_eq!(list.pop_back(), None);
// Populate list
list.push_back(1);
list.push_back(2);
list.push_back(3);
// Check normal removal
assert_eq!(list.pop_back(), Some(3));
assert_eq!(list.pop_back(), Some(2));
// Push some more just to make sure nothing's corrupted
list.push_back(4);
list.push_back(5);
// Check normal removal
assert_eq!(list.pop_back(), Some(5));
assert_eq!(list.pop_back(), Some(4));
// Check exhaustion
assert_eq!(list.pop_back(), Some(1));
assert_eq!(list.pop_back(), None);
}
#[test]
fn peek() {
let mut list = List::new();
assert!(list.peek_front().is_none());
assert!(list.peek_back().is_none());
assert!(list.peek_front_mut().is_none());
assert!(list.peek_back_mut().is_none());
list.push_front(1); list.push_front(2); list.push_front(3);
assert_eq!(&*list.peek_front().unwrap(), &3);
assert_eq!(&mut *list.peek_front_mut().unwrap(), &mut 3);
assert_eq!(&*list.peek_back().unwrap(), &1);
assert_eq!(&mut *list.peek_back_mut().unwrap(), &mut 1);
}
Are there some cases we're not testing? Probably. The combinatoric space has really blown up here. Our code is at very least not obviously wrong.
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 10 tests
test first::test::basics ... ok
test fourth::test::basics ... ok
test second::test::basics ... ok
test fourth::test::peek ... ok
test second::test::iter ... ok
test third::test::iter ... ok
test second::test::into_iter ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test third::test::basics ... ok
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured
Nice. Copy-pasting is the best kind of programming.
Iteration
Let's take a crack at iterating this bad-boy.
IntoIter
IntoIter, as always, is going to be the easiest. Just wrap the stack and
call pop:
pub struct IntoIter<T>(List<T>);
impl<T> List<T> {
pub fn into_iter(self) -> IntoIter<T> {
IntoIter(self)
}
}
impl<T> Iterator for IntoIter<T> {
type Item = T;
fn next(&mut self) -> Option<Self::Item> {
self.0.pop_front()
}
}
But we have an interesting new development. Where previously there was only ever one "natural" iteration order for our lists, a Deque is inherently bi-directional. What's so special about front-to-back? What if someone wants to iterate in the other direction?
Rust actually has an answer to this: DoubleEndedIterator. DoubleEndedIterator
inherits from Iterator (meaning all DoubleEndedIterator are Iterators) and
requires one new method: next_back. It has the exact same signature as
next, but it's supposed to yield elements from the other end. The semantics
of DoubleEndedIterator are super convenient for us: the iterator becomes a
deque. You can consume elements from the front and back until the two ends
converge, at which point the iterator is empty.
Much like Iterator and next, it turns out that next_back isn't really
something consumers of the DoubleEndedIterator really care about. Rather, the
best part of this interface is that it exposes the rev method, which wraps
up the iterator to make a new one that yields the elements in reverse order.
The semantics of this are fairly straight-forward: calls to next on the
reversed iterator are just calls to next_back.
Anyway, because we're already a deque providing this API is pretty easy:
impl<T> DoubleEndedIterator for IntoIter<T> {
fn next_back(&mut self) -> Option<T> {
self.0.pop_back()
}
}
And let's test it out:
#[test]
fn into_iter() {
let mut list = List::new();
list.push_front(1); list.push_front(2); list.push_front(3);
let mut iter = list.into_iter();
assert_eq!(iter.next(), Some(3));
assert_eq!(iter.next_back(), Some(1));
assert_eq!(iter.next(), Some(2));
assert_eq!(iter.next_back(), None);
assert_eq!(iter.next(), None);
}
cargo test
Running target/debug/lists-5c71138492ad4b4a
running 11 tests
test fourth::test::basics ... ok
test fourth::test::peek ... ok
test fourth::test::into_iter ... ok
test first::test::basics ... ok
test second::test::basics ... ok
test second::test::iter ... ok
test second::test::iter_mut ... ok
test third::test::iter ... ok
test third::test::basics ... ok
test second::test::into_iter ... ok
test second::test::peek ... ok
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured
Nice.
Iter
Iter will be a bit less forgiving. We'll have to deal with those awful Ref
things again! Because of Refs, we can't store &Nodes like we did before.
Instead, let's try to store Ref<Node>s:
pub struct Iter<'a, T>(Option<Ref<'a, Node<T>>>);
impl<T> List<T> {
pub fn iter(&self) -> Iter<T> {
Iter(self.head.as_ref().map(|head| head.borrow()))
}
}
> cargo build
So far so good. Implementing next is going to be a bit hairy, but I think
it's the same basic logic as the old stack IterMut but with extra RefCell
madness:
impl<'a, T> Iterator for Iter<'a, T> {
type Item = Ref<'a, T>;
fn next(&mut self) -> Option<Self::Item> {
self.0.take().map(|node_ref| {
self.0 = node_ref.next.as_ref().map(|head| head.borrow());
Ref::map(node_ref, |node| &node.elem)
})
}
}
cargo build
error[E0521]: borrowed data escapes outside of closure
--> src/fourth.rs:155:13
|
153 | fn next(&mut self) -> Option<Self::Item> {
| --------- `self` is declared here, outside of the closure body
154 | self.0.take().map(|node_ref| {
155 | self.0 = node_ref.next.as_ref().map(|head| head.borrow());
| ^^^^^^ -------- borrow is only valid in the closure body
| |
| reference to `node_ref` escapes the closure body here
error[E0505]: cannot move out of `node_ref` because it is borrowed
--> src/fourth.rs:156:22
|
153 | fn next(&mut self) -> Option<Self::Item> {
| --------- lifetime `'1` appears in the type of `self`
154 | self.0.take().map(|node_ref| {
155 | self.0 = node_ref.next.as_ref().map(|head| head.borrow());
| ------ -------- borrow of `node_ref` occurs here
| |
| assignment requires that `node_ref` is borrowed for `'1`
156 | Ref::map(node_ref, |node| &node.elem)
| ^^^^^^^^ move out of `node_ref` occurs here
Shoot.
node_ref doesn't live long enough. Unlike normal references, Rust doesn't let
us just split Refs up like that. The Ref we get out of head.borrow() is only
allowed to live as long as node_ref, but we end up trashing that in our
Ref::map call.
The function we want exists, and it's called [map_split][]:
pub fn map_split<U, V, F>(orig: Ref<'b, T>, f: F) -> (Ref<'b, U>, Ref<'b, V>) where
F: FnOnce(&T) -> (&U, &V),
U: ?Sized,
V: ?Sized,
Woof. Let's give it a try...
fn next(&mut self) -> Option<Self::Item> {
self.0.take().map(|node_ref| {
let (next, elem) = Ref::map_split(node_ref, |node| {
(&node.next, &node.elem)
});
self.0 = next.as_ref().map(|head| head.borrow());
elem
})
}
cargo build
Compiling lists v0.1.0 (/Users/ABeingessner/dev/temp/lists)
error[E0521]: borrowed data escapes outside of closure
--> src/fourth.rs:159:13
|
153 | fn next(&mut self) -> Option<Self::Item> {
| --------- `self` is declared here, outside of the closure body
...
159 | self.0 = next.as_ref().map(|head| head.borrow());
| ^^^^^^ ---- borrow is only valid in the closure body
| |
| reference to `next` escapes the closure body here
Ergh. We need to Ref::Map again to get our lifetimes right. But Ref::Map
returns a Ref and we need an Option<Ref>, but we need to go through the
Ref to map over our Option...
stares into distance for a long time
??????
fn next(&mut self) -> Option<Self::Item> {
self.0.take().map(|node_ref| {
let (next, elem) = Ref::map_split(node_ref, |node| {
(&node.next, &node.elem)
});
self.0 = if next.is_some() {
Some(Ref::map(next, |next| &**next.as_ref().unwrap()))
} else {
None
};
elem
})
}
error[E0308]: mismatched types
--> src/fourth.rs:162:22
|
162 | Some(Ref::map(next, |next| &**next.as_ref().unwrap()))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `fourth::Node`, found struct `std::cell::RefCell`
|
= note: expected type `std::cell::Ref<'_, fourth::Node<_>>`
found type `std::cell::Ref<'_, std::cell::RefCell<fourth::Node<_>>>`
Oh. Right. There's multiple RefCells. The deeper we walk into the list, the more nested we become under each RefCell. We would need to maintain, like, a stack of Refs to represent all the outstanding loans we're holding, because if we stop looking at an element we need to decrement the borrow-count on every RefCell that comes before it.................
I don't think there's anything we can do here. It's a dead end. Let's try getting out of the RefCells.
What about our Rcs. Who said we even needed to store references?
Why can't we just Clone the whole Rc to get a nice owning handle into the middle
of the list?
pub struct Iter<T>(Option<Rc<Node<T>>>);
impl<T> List<T> {
pub fn iter(&self) -> Iter<T> {
Iter(self.head.as_ref().map(|head| head.clone()))
}
}
impl<T> Iterator for Iter<T> {
type Item =
Uh... Wait what do we return now? &T? Ref<T>?
No, none of those work... our Iter doesn't have a lifetime anymore! Both &T
and Ref<T> require us to declare some lifetime up front before we get into
next. But anything we manage to get out of our Rc would be borrowing the
Iterator... brain... hurt... aaaaaahhhhhh
Maybe we can... map... the Rc... to get an Rc<T>? Is that a thing? Rc's docs
don't seem to have anything like that. Actually someone made a crate
that lets you do that.
But wait, even if we do that then we've got an even bigger problem: the
dreaded spectre of iterator invalidation. Previously we've been totally immune
to iterator invalidation, because the Iter borrowed the list, leaving it totally
immutable. However if our Iter was yielding Rcs, they wouldn't borrow the list
at all! That means people can start calling push and pop on the list while
they hold pointers into it!
Oh lord, what will that do?!
Well, pushing is actually fine. We've got a view into some sub-range of the list, and the list will just grow beyond our sights. No biggie.
However pop is another story. If they're popping elements outside of our
range, it should still be fine. We can't see those nodes so nothing will
happen. However if they try to pop off the node we're pointing at... everything
will blow up! In particular when they go to unwrap the result of the
try_unwrap, it will actually fail, and the whole program will panic.
That's actually pretty cool. We can get tons of interior owning pointers into the list and mutate it at the same time and it will just work until they try to remove the nodes that we're pointing at. And even then we don't get dangling pointers or anything, the program will deterministically panic!
But having to deal with iterator invalidation on top of mapping Rcs just
seems... bad. Rc<RefCell> has really truly finally failed us. Interestingly,
we've experienced an inversion of the persistent stack case. Where the
persistent stack struggled to ever reclaim ownership of the data but could get
references all day every day, our list had no problem gaining ownership, but
really struggled to loan our references.
Although to be fair, most of our struggles revolved around wanting to hide the implementation details and have a decent API. We could do everything fine if we wanted to just pass around Nodes all over the place.
Heck, we could make multiple concurrent IterMuts that were runtime checked to not be mutable accessing the same element!
Really, this design is more appropriate for an internal data structure that never makes it out to consumers of the API. Interior mutability is great for writing safe applications. Not so much safe libraries.
Anyway, that's me giving up on Iter and IterMut. We could do them, but ugh.
Final Code
Alright, so that's implementing a 100% safe doubly-linked list in Rust. It was a nightmare to implement, leaks implementation details, and doesn't support several fundamental operations.
But, it exists.
Oh, I guess it's also riddled with tons of "unnecessary" runtime checks for
correctness between Rc and RefCell. I put unnecessary in quotes because
they're actually necessary to guarantee the whole actually being safe thing.
We encountered a few places where those checks actually were necessary.
Doubly-linked lists have a horribly tangled aliasing and ownership story!
Still, it's a thing we can do. Especially if we don't care about exposing internal data structures to our consumers.
From here on out, we're going to be focusing on other side of this coin: getting back all the control by making our implementation unsafe.
#![allow(unused)] fn main() { use std::rc::Rc; use std::cell::{Ref, RefMut, RefCell}; pub struct List<T> { head: Link<T>, tail: Link<T>, } type Link<T> = Option<Rc<RefCell<Node<T>>>>; struct Node<T> { elem: T, next: Link<T>, prev: Link<T>, } impl<T> Node<T> { fn new(elem: T) -> Rc<RefCell<Self>> { Rc::new(RefCell::new(Node { elem: elem, prev: None, next: None, })) } } impl<T> List<T> { pub fn new() -> Self { List { head: None, tail: None } } pub fn push_front(&mut self, elem: T) { let new_head = Node::new(elem); match self.head.take() { Some(old_head) => { old_head.borrow_mut().prev = Some(new_head.clone()); new_head.borrow_mut().next = Some(old_head); self.head = Some(new_head); } None => { self.tail = Some(new_head.clone()); self.head = Some(new_head); } } } pub fn push_back(&mut self, elem: T) { let new_tail = Node::new(elem); match self.tail.take() { Some(old_tail) => { old_tail.borrow_mut().next = Some(new_tail.clone()); new_tail.borrow_mut().prev = Some(old_tail); self.tail = Some(new_tail); } None => { self.head = Some(new_tail.clone()); self.tail = Some(new_tail); } } } pub fn pop_back(&mut self) -> Option<T> { self.tail.take().map(|old_tail| { match old_tail.borrow_mut().prev.take() { Some(new_tail) => { new_tail.borrow_mut().next.take(); self.tail = Some(new_tail); } None => { self.head.take(); } } Rc::try_unwrap(old_tail).ok().unwrap().into_inner().elem }) } pub fn pop_front(&mut self) -> Option<T> { self.head.take().map(|old_head| { match old_head.borrow_mut().next.take() { Some(new_head) => { new_head.borrow_mut().prev.take(); self.head = Some(new_head); } None => { self.tail.take(); } } Rc::try_unwrap(old_head).ok().unwrap().into_inner().elem }) } pub fn peek_front(&self) -> Option<Ref<T>> { self.head.as_ref().map(|node| { Ref::map(node.borrow(), |node| &node.elem) }) } pub fn peek_back(&self) -> Option<Ref<T>> { self.tail.as_ref().map(|node| { Ref::map(node.borrow(), |node| &node.elem) }) } pub fn peek_back_mut(&mut self) -> Option<RefMut<T>> { self.tail.as_ref().map(|node| { RefMut::map(node.borrow_mut(), |node| &mut node.elem) }) } pub fn peek_front_mut(&mut self) -> Option<RefMut<T>> { self.head.as_ref().map(|node| { RefMut::map(node.borrow_mut(), |node| &mut node.elem) }) } pub fn into_iter(self) -> IntoIter<T> { IntoIter(self) } } impl<T> Drop for List<T> { fn drop(&mut self) { while self.pop_front().is_some() {} } } pub struct IntoIter<T>(List<T>); impl<T> Iterator for IntoIter<T> { type Item = T; fn next(&mut self) -> Option<T> { self.0.pop_front() } } impl<T> DoubleEndedIterator for IntoIter<T> { fn next_back(&mut self) -> Option<T> { self.0.pop_back() } } #[cfg(test)] mod test { use super::List; #[test] fn basics() { let mut list = List::new(); // Check empty list behaves right assert_eq!(list.pop_front(), None); // Populate list list.push_front(1); list.push_front(2); list.push_front(3); // Check normal removal assert_eq!(list.pop_front(), Some(3)); assert_eq!(list.pop_front(), Some(2)); // Push some more just to make sure nothing's corrupted list.push_front(4); list.push_front(5); // Check normal removal assert_eq!(list.pop_front(), Some(5)); assert_eq!(list.pop_front(), Some(4)); // Check exhaustion assert_eq!(list.pop_front(), Some(1)); assert_eq!(list.pop_front(), None); // ---- back ----- // Check empty list behaves right assert_eq!(list.pop_back(), None); // Populate list list.push_back(1); list.push_back(2); list.push_back(3); // Check normal removal assert_eq!(list.pop_back(), Some(3)); assert_eq!(list.pop_back(), Some(2)); // Push some more just to make sure nothing's corrupted list.push_back(4); list.push_back(5); // Check normal removal assert_eq!(list.pop_back(), Some(5)); assert_eq!(list.pop_back(), Some(4)); // Check exhaustion assert_eq!(list.pop_back(), Some(1)); assert_eq!(list.pop_back(), None); } #[test] fn peek() { let mut list = List::new(); assert!(list.peek_front().is_none()); assert!(list.peek_back().is_none()); assert!(list.peek_front_mut().is_none()); assert!(list.peek_back_mut().is_none()); list.push_front(1); list.push_front(2); list.push_front(3); assert_eq!(&*list.peek_front().unwrap(), &3); assert_eq!(&mut *list.peek_front_mut().unwrap(), &mut 3); assert_eq!(&*list.peek_back().unwrap(), &1); assert_eq!(&mut *list.peek_back_mut().unwrap(), &mut 1); } #[test] fn into_iter() { let mut list = List::new(); list.push_front(1); list.push_front(2); list.push_front(3); let mut iter = list.into_iter(); assert_eq!(iter.next(), Some(3)); assert_eq!(iter.next_back(), Some(1)); assert_eq!(iter.next(), Some(2)); assert_eq!(iter.next_back(), None); assert_eq!(iter.next(), None); } } }
An Ok Unsafe Singly-Linked Queue
Ok that reference-counted interior mutability stuff got a little out of control. Surely Rust doesn't really expect you to do that sort of thing in general? Well, yes and no. Rc and Refcell can be great for handling simple cases, but they can get unwieldy. Especially if you want to hide that it's happening. There's gotta be a better way!
In this chapter we're going to roll back to singly-linked lists and implement a singly-linked queue to dip our toes into raw pointers and Unsafe Rust.
NARRATOR: And I will point out the mistakes.
And we won't make any mistakes.
Let's add a new file called fifth.rs:
// in lib.rs
pub mod first;
pub mod second;
pub mod third;
pub mod fourth;
pub mod fifth;
Our code is largely going to be derived from second.rs, since a queue is mostly an augmentation of a stack in the world of linked lists. Still, we're going to go from scratch because there's some fundamental issues we want to address with layout and what-not.
Layout
So what's a singly-linked queue like? Well, when we had a singly-linked stack we pushed onto one end of the list, and then popped off the same end. The only difference between a stack and a queue is that a queue pops off the other end. So from our stack implementation we have:
input list:
[Some(ptr)] -> (A, Some(ptr)) -> (B, None)
stack push X:
[Some(ptr)] -> (X, Some(ptr)) -> (A, Some(ptr)) -> (B, None)
stack pop:
[Some(ptr)] -> (A, Some(ptr)) -> (B, None)
To make a queue, we just need to decide which operation to move to the end of the list: push, or pop? Since our list is singly-linked, we can actually move either operation to the end with the same amount of effort.
To move push to the end, we just walk all the way to the None and set it
to Some with the new element.
input list:
[Some(ptr)] -> (A, Some(ptr)) -> (B, None)
flipped push X:
[Some(ptr)] -> (A, Some(ptr)) -> (B, Some(ptr)) -> (X, None)
To move pop to the end, we just walk all the way to the node before the
None, and take it:
input list:
[Some(ptr)] -> (A, Some(ptr)) -> (B, Some(ptr)) -> (X, None)
flipped pop:
[Some(ptr)] -> (A, Some(ptr)) -> (B, None)
We could do this today and call it quits, but that would stink! Both of these operations walk over the entire list. Some would argue that such a queue implementation is indeed a queue because it exposes the right interface. However I believe that performance guarantees are part of the interface. I don't care about precise asymptotic bounds, just "fast" vs "slow". Queues guarantee that push and pop are fast, and walking over the whole list is definitely not fast.
One key observation is that we're wasting a ton of work doing the same thing over and over. Can we "cache" all that work and reuse it? Why, yes! We can store a pointer to the end of the list, and just jump straight to there!
It turns out that only one inversion of push and pop works with this.
To invert pop we would have to move the "tail" pointer backwards, but
because our list is singly-linked, we can't do that efficiently.
If we instead invert push we only have to move the "head" pointer
forward, which is easy.
Let's try that:
use std::mem;
pub struct List<T> {
head: Link<T>,
tail: Link<T>, // NEW!
}
type Link<T> = Option<Box<Node<T>>>;
struct Node<T> {
elem: T,
next: Link<T>,
}
impl<T> List<T> {
pub fn new() -> Self {
List { head: None, tail: None }
}
pub fn push(&mut self, elem: T) {
let new_tail = Box::new(Node {
elem: elem,
// When you push onto the tail, your next is always None
next: None,
});
// swap the old tail to point to the new tail
let old_tail = mem::replace(&mut self.tail, Some(new_tail));
match old_tail {
Some(mut old_tail) => {
// If the old tail existed, update it to point to the new tail
old_tail.next = Some(new_tail);
}
None => {
// Otherwise, update the head to point to it
self.head = Some(new_tail);
}
}
}
}
I'm going a bit faster with the impl details now since we should be pretty
comfortable with this sort of thing. Not that you should necessarily expect
to produce this code on the first try. I'm just skipping over some of the
trial-and-error we've had to deal with before. I actually made a ton of mistakes
writing this code that I'm not showing, but you can only see me leave off a mut or
; so many times before it stops being instructive. Don't worry, we'll see
plenty of other error messages!
> cargo build
error[E0382]: use of moved value: `new_tail`
--> src/fifth.rs:38:38
|
26 | let new_tail = Box::new(Node {
| -------- move occurs because `new_tail` has type `std::boxed::Box<fifth::Node<T>>`, which does not implement the `Copy` trait
...
33 | let old_tail = mem::replace(&mut self.tail, Some(new_tail));
| -------- value moved here
...
38 | old_tail.next = Some(new_tail);
| ^^^^^^^^ value used here after move
Shoot!
use of moved value:
new_tail
Box doesn't implement Copy, so we can't just assign it to two locations. More
importantly, Box owns the thing it points to, and will try to free it when
it's dropped. If our push implementation compiled, we'd double-free the tail
of our list! Actually, as written, our code would free the old_tail on every
push. Yikes! 🙀
Alright, well we know how to make a non-owning pointer. That's just a reference!
pub struct List<T> {
head: Link<T>,
tail: Option<&mut Node<T>>, // NEW!
}
type Link<T> = Option<Box<Node<T>>>;
struct Node<T> {
elem: T,
next: Link<T>,
}
impl<T> List<T> {
pub fn new() -> Self {
List { head: None, tail: None }
}
pub fn push(&mut self, elem: T) {
let new_tail = Box::new(Node {
elem: elem,
// When you push onto the tail, your next is always None
next: None,
});
// Put the box in the right place, and then grab a reference to its Node
let new_tail = match self.tail.take() {
Some(old_tail) => {
// If the old tail existed, update it to point to the new tail
old_tail.next = Some(new_tail);
old_tail.next.as_deref_mut()
}
None => {
// Otherwise, update the head to point to it
self.head = Some(new_tail);
self.head.as_deref_mut()
}
};
self.tail = new_tail;
}
}
Nothing too tricky here. Same basic idea as the previous code, except we're using some of that implicit return goodness to extract the tail reference from wherever we stuff the actual Box.
> cargo build
error[E0106]: missing lifetime specifier
--> src/fifth.rs:3:18
|
3 | tail: Option<&mut Node<T>>, // NEW!
| ^ expected lifetime parameter
Oh right, we need to give lifetimes to references in types. Hmm... what's the
lifetime of this reference? Well, this seems like IterMut, right? Let's try
what we did for IterMut, and just add a generic 'a:
pub struct List<'a, T> {
head: Link<T>,
tail: Option<&'a mut Node<T>>, // NEW!
}
type Link<T> = Option<Box<Node<T>>>;
struct Node<T> {
elem: T,
next: Link<T>,
}
impl<'a, T> List<'a, T> {
pub fn new() -> Self {
List { head: None, tail: None }
}
pub fn push(&mut self, elem: T) {
let new_tail = Box::new(Node {
elem: elem,
// When you push onto the tail, your next is always None
next: None,
});
// Put the box in the right place, and then grab a reference to its Node
let new_tail = match self.tail.take() {
Some(old_tail) => {
// If the old tail existed, update it to point to the new tail
old_tail.next = Some(new_tail);
old_tail.next.as_deref_mut()
}
None => {
// Otherwise, update the head to point to it
self.head = Some(new_tail);
self.head.as_deref_mut()
}
};
self.tail = new_tail;
}
}
cargo build
error[E0495]: cannot infer an appropriate lifetime for autoref due to conflicting requirements
--> src/fifth.rs:35:27
|
35 | self.head.as_deref_mut()
| ^^^^^^^^^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 18:5...
--> src/fifth.rs:18:5
|
18 | / pub fn push(&mut self, elem: T) {
19 | | let new_tail = Box::new(Node {
20 | | elem: elem,
21 | | // When you push onto the tail, your next is always None
... |
39 | | self.tail = new_tail;
40 | | }
| |_____^
note: ...so that reference does not outlive borrowed content
--> src/fifth.rs:35:17
|
35 | self.head.as_deref_mut()
| ^^^^^^^^^
note: but, the lifetime must be valid for the lifetime 'a as defined on the impl at 13:6...
--> src/fifth.rs:13:6
|
13 | impl<'a, T> List<'a, T> {
| ^^
= note: ...so that the expression is assignable:
expected std::option::Option<&'a mut fifth::Node<T>>
found std::option::Option<&mut fifth::Node<T>>
Woah, that's a really detailed error message. That's a bit concerning, because it suggests we're doing something really messed up. Here's an interesting part:
the lifetime must be valid for the lifetime
'aas defined on the impl
We're borrowing from self, but the compiler wants us to last as long as 'a,
what if we tell it self does last that long..?
pub fn push(&'a mut self, elem: T) {
cargo build
warning: field is never used: `elem`
--> src/fifth.rs:9:5
|
9 | elem: T,
| ^^^^^^^
|
= note: #[warn(dead_code)] on by default
Oh, hey, that worked! Great!
Let's just do pop too:
pub fn pop(&'a mut self) -> Option<T> {
// Grab the list's current head
self.head.take().map(|head| {
let head = *head;
self.head = head.next;
// If we're out of `head`, make sure to set the tail to `None`.
if self.head.is_none() {
self.tail = None;
}
head.elem
})
}
And write a quick test for that:
#[cfg(test)]
mod test {
use super::List;
#[test]
fn basics() {
let mut list = List::new();
// Check empty list behaves right
assert_eq!(list.pop(), None);
// Populate list
list.push(1);
list.push(2);
list.push(3);
// Check normal removal
assert_eq!(list.pop(), Some(1));
assert_eq!(list.pop(), Some(2));
// Push some more just to make sure nothing's corrupted
list.push(4);
list.push(5);
// Check normal removal
assert_eq!(list.pop(), Some(3));
assert_eq!(list.pop(), Some(4));
// Check exhaustion
assert_eq!(list.pop(), Some(5));
assert_eq!(list.pop(), None);
}
}
cargo test
error[E0499]: cannot borrow `list` as mutable more than once at a time
--> src/fifth.rs:68:9
|
65 | assert_eq!(list.pop(), None);
| ---- first mutable borrow occurs here
...
68 | list.push(1);
| ^^^^
| |
| second mutable borrow occurs here
| first borrow later used here
error[E0499]: cannot borrow `list` as mutable more than once at a time
--> src/fifth.rs:69:9
|
65 | assert_eq!(list.pop(), None);
| ---- first mutable borrow occurs here
...
69 | list.push(2);
| ^^^^
| |
| second mutable borrow occurs here
| first borrow later used here
error[E0499]: cannot borrow `list` as mutable more than once at a time
--> src/fifth.rs:70:9
|
65 | assert_eq!(list.pop(), None);
| ---- first mutable borrow occurs here
...
70 | list.push(3);
| ^^^^
| |
| second mutable borrow occurs here
| first borrow later used here
....
** WAY MORE LINES OF ERRORS **
....
error: aborting due to 11 previous errors
🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀🙀
Oh my goodness.
The compiler's not wrong for vomiting all over us. We just committed a
cardinal Rust sin: we stored a reference to ourselves inside ourselves.
Somehow, we managed to convince Rust that this totally made sense in our
push and pop implementations (I was legitimately shocked we did).
The reason this sort of works is that Rust doesn't really have the notion of a pointer into yourself at all. Each part of the code is technically correct in isolation (we can call push and pop once) but then the absurdity of what we created takes affect and everything just locks up.
I'm sure there is some use for what we've written, but as far as I'm concerned it's
just syntatically valid gibberish. We're saying we contain something with
lifetime 'a, and that push and pop borrows self for that lifetime.
That's weird but Rust can look at each part of our code individually and
it doesn't see any rules being broken.
But as soon as we try to actually use the list, the compiler quickly goes
"yep you've borrowed self mutably for 'a, so you can't use self anymore
until the end of 'a" but also "because you contain 'a, it must be valid
for the entire list's existence".
It's nearly a contradiction but there is one solution: as soon as you push
or pop, the list "pins" itself in place and can't be accessed anymore. It has
swallowed its own proverbial tail, and ascended to a world of dreams.
NARRATOR: it didn't exist when this book was first written, but Rust actually formalized the notion of a pin into something useful! This was probably the most complex addition to the language since the borrowchecker. We don't want our list to be pinned though!
Pins are necessary and useful for async-await/futures/coroutines because the compiler needs to be able to bundle up all the local variables of a function into some kind of struct and store them somewhere until the future/coroutine is ready to be resumed. Since local variables can reference other local variables, and we want that to work, these structs can end up containing references to themselves!
So to
awaitoryieldRust needs a way to be able to properly describe and manipulate pinned values. Thankfully all of this stuff is largely just hidden away in automatic compiler machinery and no one actually has to think aboutPin(or even Futures) under normal circumstances. The main exception is that this stuff is very important for the folks building and designing async runtimes like tokio.We will not be implementing an async runtime in this book. I know my friends know all sorts of "cool" (messed up) tricks you can do with
Pin, but from what I can tell, I'd be happier to just not know them. I will continue to tell myself that Pinned types aren't real and they can't hurt me.
Our pop implementation hints at why storing a reference to ourselves
inside ourselves could be really dangerous:
// ...
if self.head.is_none() {
self.tail = None;
}
What if we forgot to do this? Then our tail would point to some node that had been removed from the list. Such a node would be instantly freed, and we'd have a dangling pointer which Rust was supposed to protect us from!
And indeed Rust is protecting us from that kind of danger. Just in a very... roundabout way.
So what can we do? Go back to Rc<RefCell>> hell?
Please. No.
No, instead we're going to go off the rails and use raw pointers. Our layout is going to look like this:
pub struct List<T> {
head: Link<T>,
tail: *mut Node<T>, // DANGER DANGER
}
type Link<T> = Option<Box<Node<T>>>;
struct Node<T> {
elem: T,
next: Link<T>,
}
And that's that. None of this wimpy reference-counted-dynamic-borrow-checking nonsense! Real. Hard. Unchecked. Pointers.
NARRATOR: This implementation was in fact still dangerously wrong, but it wasn't yet time to learn that lesson. The next section will learn that the hard way, as usual.
Let's be C everyone. Let's be C all day.
I'm home. I'm ready.
Hello unsafe.
NARRATOR: Wow, just incredible hubris from the author here.
Unsafe Rust
This is a serious, big, complicated, and dangerous topic. It's so serious that I wrote an entire other book on it.
The long and the short of it is that every language is actually unsafe as soon as you allow calling into other languages, because you can just have C do arbitrarily bad things. Yes: Java, Python, Ruby, Haskell... everyone is wildly unsafe in the face of Foreign Function Interfaces (FFI).
Rust embraces this truth by splitting itself into two languages: Safe Rust, and Unsafe Rust. So far we've only worked with Safe Rust. It's completely 100% safe... except that it can FFI into Unsafe Rust.
Unsafe Rust is a superset of Safe Rust. It's completely the same as Safe Rust in all its semantics and rules, you're just allowed to do a few extra things that are wildly unsafe and can cause the dreaded Undefined Behaviour that haunts C.
Again, this is a really huge topic that has a lot of interesting corner cases. I really don't want to go really deep into it (well, I do. I did. Read that book). That's ok, because with linked lists we can actually ignore almost all of it.
NARRATOR: This was a lie, but it did seem true in 2015.
The main Unsafe tool we'll be using are raw pointers. Raw pointers are basically C's pointers. They have no inherent aliasing rules. They have no lifetimes. They can be null. They can be misaligned. They can be dangling. They can point to uninitialized memory. They can be cast to and from integers. They can be cast to point to a different type. Mutability? Cast it. Pretty much everything goes, and that means pretty much anything can go wrong.
NARRATOR: no inherent aliasing rules, eh? Ah, the innocence of youth.
This is some bad stuff and honestly you'll live a happier life never having to touch these. Unfortunately, we want to write linked lists, and linked lists are awful. That means we're going to have to use unsafe pointers.
There are two kinds of raw pointer: *const T and *mut T. These are meant to
be const T* and T* from C, but we really don't care about what C thinks they
mean that much. You can only dereference a *const T to an &T, but much like
the mutability of a variable, this is just a lint against incorrect usage. At
most it just means you have to cast the *const to a *mut first. Although if
you don't actually have permission to mutate the referent of the pointer,
you're gonna have a bad time.
Anyway, we'll get a better feel for this as we write some code. For now,
*mut T == &unchecked mut T!
Basics
NARRATOR: This section has a looming fundamental error in it, because that's the whole point of the book. However once we start using
unsafeit's possible to do things wrong and still have everything compile and seemingly work. The fundamental mistake will be identified in the next section. Don't actually use the contents of this section in production code!
Alright, back to basics. How do we construct our list?
Before we just did:
impl<T> List<T> {
pub fn new() -> Self {
List { head: None, tail: None }
}
}
But we're not using Option for the tail anymore:
> cargo build
error[E0308]: mismatched types
--> src/fifth.rs:15:34
|
15 | List { head: None, tail: None }
| ^^^^ expected *-ptr, found
| enum `std::option::Option`
|
= note: expected type `*mut fifth::Node<T>`
found type `std::option::Option<_>`
We could use an Option, but unlike Box, *mut is nullable. This means it
can't benefit from the null pointer optimization. Instead, we'll be using null
to represent None.
So how do we get a null pointer? There's a few ways, but I prefer to use
std::ptr::null_mut(). If you want, you can also use 0 as *mut _, but that
just seems so messy.
use std::ptr;
// defns...
impl<T> List<T> {
pub fn new() -> Self {
List { head: None, tail: ptr::null_mut() }
}
}
cargo build
warning: field is never used: `head`
--> src/fifth.rs:4:5
|
4 | head: Link<T>,
| ^^^^^^^^^^^^^
|
= note: #[warn(dead_code)] on by default
warning: field is never used: `tail`
--> src/fifth.rs:5:5
|
5 | tail: *mut Node<T>,
| ^^^^^^^^^^^^^^^^^^
warning: field is never used: `elem`
--> src/fifth.rs:11:5
|
11 | elem: T,
| ^^^^^^^
warning: field is never used: `head`
--> src/fifth.rs:12:5
|
12 | head: Link<T>,
| ^^^^^^^^^^^^^
shush compiler, we will use them soon.
Alright, let's move on to writing push again. This time, instead of grabbing
an Option<&mut Node<T>> after we insert, we're just going to grab a
*mut Node<T> to the insides of the Box right away. We know we can soundly do
this because the contents of a Box has a stable address, even if we move the
Box around. Of course, this isn't safe, because if we just drop the Box we'll
have a pointer to freed memory.
How do we make a raw pointer from a normal pointer? Coercions! If a variable is declared to be a raw pointer, a normal reference will coerce into it:
let raw_tail: *mut _ = &mut *new_tail;
We have all the info we need. We can translate our code into, approximately, the previous reference version:
pub fn push(&mut self, elem: T) {
let mut new_tail = Box::new(Node {
elem: elem,
next: None,
});
let raw_tail: *mut _ = &mut *new_tail;
// .is_null checks for null, equivalent to checking for None
if !self.tail.is_null() {
// If the old tail existed, update it to point to the new tail
self.tail.next = Some(new_tail);
} else {
// Otherwise, update the head to point to it
self.head = Some(new_tail);
}
self.tail = raw_tail;
}
> cargo build
error[E0609]: no field `next` on type `*mut fifth::Node<T>`
--> src/fifth.rs:31:23
|
31 | self.tail.next = Some(new_tail);
| ----------^^^^
| |
| help: `self.tail` is a raw pointer;
| try dereferencing it: `(*self.tail).next`
Huh? We have a pointer to a Node, why can't we get the next field?
Rust is kinda a jerk when you use raw pointers. To access the contents of a raw pointer, it insists that we manually deref them, because it's such an unsafe operation. So let's do that:
*self.tail.next = Some(new_tail);
> cargo build
error[E0609]: no field `next` on type `*mut fifth::Node<T>`
--> src/fifth.rs:31:23
|
31 | *self.tail.next = Some(new_tail);
| -----------^^^^
| |
| help: `self.tail` is a raw pointer;
| try dereferencing it: `(*self.tail).next`
Uuuugh operator precedence.
(*self.tail).next = Some(new_tail);
> cargo build
error[E0133]: dereference of raw pointer is unsafe and requires
unsafe function or block
--> src/fifth.rs:31:13
|
31 | (*self.tail).next = Some(new_tail);
| ^^^^^^^^^^^^^^^^^ dereference of raw pointer
|
= note: raw pointers may be NULL, dangling or unaligned;
they can violate aliasing rules and cause data races:
all of these are undefined behavior
THIS. SHOULDN'T. BE. THIS. HARD.
Remember how I said Unsafe Rust is like an FFI language for Safe Rust? Well, the
compiler wants us to explicitly delimit where we're doing this FFI-ing. We have
two options. First, we can mark our entire function as unsafe, in which case
it becomes an Unsafe Rust function and can only be called in an unsafe
context. This isn't great, because we want our list to be safe to use. Second,
we can write an unsafe block inside our function, to delimit the FFI boundary.
This declares the overall function to be safe. Let's do that one:
pub fn push(&mut self, elem: T) {
let mut new_tail = Box::new(Node {
elem: elem,
next: None,
});
let raw_tail: *mut _ = &mut *new_tail;
if !self.tail.is_null() {
// Hello Compiler, I Know I Am Doing Something Dangerous And
// I Promise To Be A Good Programmer Who Never Makes Mistakes.
unsafe {
(*self.tail).next = Some(new_tail);
}
} else {
self.head = Some(new_tail);
}
self.tail = raw_tail;
}
> cargo build
warning: field is never used: `elem`
--> src/fifth.rs:11:5
|
11 | elem: T,
| ^^^^^^^
|
= note: #[warn(dead_code)] on by default
Yay!
It's kind've interesting that that's the only place we've had to write an unsafe block so far. We do raw pointer stuff all over the place, what's up with that?
It turns out that Rust is a massive rules-lawyer pedant when it comes to
unsafe. We quite reasonably want to maximize the set of Safe Rust programs,
because those are programs we can be much more confident in. To accomplish this,
Rust carefully carves out a minimal surface area for unsafety. Note that all
the other places we've worked with raw pointers has been assigning them, or
just observing whether they're null or not.
If you never actually dereference a raw pointer those are totally safe things to do. You're just reading and writing an integer! The only time you can actually get into trouble with a raw pointer is if you actually dereference it. So Rust says only that operation is unsafe, and everything else is totally safe.
Super. Pedantic. But technically correct.
NARRATOR: Somewhere on the other side of the world, a hardware engineer feels a shiver down her spine — someone must be insisting pointers are just integers again. She looks down at her proposal for a new hardware pointer authentication scheme and sheds a single tear. The compiler engineer next door feels nothing — they long ago learned to always wear a heavy sweater.
Having only some of the pointer operations be actually unsafe raises an
interesting problem: although we're supposed to delimit the scope of the
unsafety with the unsafe block, it actually depends on state that was
established outside of the block. Outside of the function, even!
This is what I call unsafe taint. As soon as you use unsafe in a module,
that whole module is tainted with unsafety. Everything has to be correctly
written in order to make sure all invariants are upheld for the unsafe code.
This taint is manageable because of privacy. Outside of our module, all of our struct fields are totally private, so no one else can mess with our state in arbitrary ways. As long as no combination of the APIs we expose causes bad stuff to happen, as far as an outside observer is concerned, all of our code is safe! And really, this is no different from the FFI case. No one needs to care if some python math library shells out to C as long as it exposes a safe interface.
Anyway, let's move on to pop, which is pretty much verbatim the reference
version:
pub fn pop(&mut self) -> Option<T> {
self.head.take().map(|head| {
let head = *head;
self.head = head.next;
if self.head.is_none() {
self.tail = ptr::null_mut();
}
head.elem
})
}
Again we see another case where safety is stateful. If we fail to null out the
tail pointer in this function, we'll see no problems at all. However
subsequent calls to push will start writing to the dangling tail!
Let's test it out:
#[cfg(test)]
mod test {
use super::List;
#[test]
fn basics() {
let mut list = List::new();
// Check empty list behaves right
assert_eq!(list.pop(), None);
// Populate list
list.push(1);
list.push(2);
list.push(3);
// Check normal removal
assert_eq!(list.pop(), Some(1));
assert_eq!(list.pop(), Some(2));
// Push some more just to make sure nothing's corrupted
list.push(4);
list.push(5);
// Check normal removal
assert_eq!(list.pop(), Some(3));
assert_eq!(list.pop(), Some(4));
// Check exhaustion
assert_eq!(list.pop(), Some(5));
assert_eq!(list.pop(), None);
// Check the exhaustion case fixed the pointer right
list.push(6);
list.push(7);
// Check normal removal
assert_eq!(list.pop(), Some(6));
assert_eq!(list.pop(), Some(7));
assert_eq!(list.pop(), None);
}
}
This is just the stack test, but with the expected pop results flipped around.
I also added some extra steps at the end to make sure that tail-pointer
corruption case in pop doesn't occur.
cargo test
running 12 tests
test fifth::test::basics ... ok
test first::test::basics ... ok
test fourth::test::basics ... ok
test fourth::test::peek ... ok
test second::test::basics ... ok
test fourth::test::into_iter ... ok
test second::test::into_iter ... ok
test second::test::iter ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test third::test::basics ... ok
test third::test::iter ... ok
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured
Gold Star!
NARRATOR: Here it comes...
Miri
nervously laughs This unsafe stuff is so easy, I don't know why everyone says otherwise. Our program works perfectly.
NARRATOR: 🙂
...right?
NARRATOR: 🙂
Well, we're writing unsafe code now, so the compiler can't help us catch mistakes as well. It could be that the tests happened to work, but were actually doing something non-deterministic. Something Undefined Behavioury.
But what can we do? We've pried open the windows and snuck out of rustc's classroom. No one can help us now.
...Wait, who's that sketchy looking person in the alleyway?
"Hey kid, you wanna interpret some Rust code?"
Wh- no? Why,
"It's wild man, it can validate that the actual dynamic execution of your program conforms to the semantics of Rust's memory model. Blows your mind..."
What?
"It checks if you Do An Undefined Behaviour."
I guess I could try interpretters just once.
"You've got rustup installed right?"
Of course I do, it's the tool for having an up to date Rust toolchain!
> rustup +nightly-2022-01-21 component add miri
info: syncing channel updates for 'nightly-2022-01-21-x86_64-pc-windows-msvc'
info: latest update on 2022-01-21, rust version 1.60.0-nightly (777bb86bc 2022-01-20)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
info: installing component 'rust-std'
info: installing component 'rustc'
info: installing component 'rustfmt'
info: downloading component 'miri'
info: installing component 'miri'
What did you just install on my computer!?
"The Good Stuff"
NARRATOR: Some weird stuff going on with toolchain versions:
The tool we're installing,
miri, works closely with rustc's internals, so it's only available for nightly toolchains.
+nightly-2022-01-21tellsrustupwe want to install miri with the rust nightly toolchain for that date. I'm giving a specific date because sometimes miri falls behind and can't be built for a few nightlies. rustup will automatically download whatever toolchain we specify with+if we don't have it installed yet.2022-01-21 is just a nightly I know has miri support, which you can check on this status page. You can just use
+nightlyif you're feeling lucky.Whenever we invoke miri via
cargo miriwe will also use this+syntax to specify the toolchain we installed miri on. If you don't want to have to specify it every time, you can userustup override set.
> cargo +nightly-2022-01-21 miri test
I will run `"cargo.exe" "install" "xargo"` to install
a recent enough xargo. Proceed? [Y/n]
UH WHAT ON EARTH IS XARGO?
"It's fine, don't worry about it."
> y
Updating crates.io index
Installing xargo v0.3.24
...
Finished release [optimized] target(s) in 10.65s
Installing C:\Users\ninte\.cargo\bin\xargo-check.exe
Installing C:\Users\ninte\.cargo\bin\xargo.exe
Installed package `xargo v0.3.24` (executables `xargo-check.exe`, `xargo.exe`)
I will run `"rustup" "component" "add" "rust-src"` to install
the `rust-src` component for the selected toolchain. Proceed? [Y/n]
UH???
"Who doesn't love having a copy of Rust's source code?"
> y
info: downloading component 'rust-src'
info: installing component 'rust-src'
"Aw yeah it's ready, here's the good part."
Compiling lists v0.1.0 (C:\Users\ninte\dev\tmp\lists)
Finished test [unoptimized + debuginfo] target(s) in 0.25s
Running unittests (lists-5cc11d9ee5c3e924.exe)
error: Undefined Behavior: trying to reborrow for Unique at alloc84055,
but parent tag <209678> does not have an appropriate item in
the borrow stack
--> \lib\rustlib\src\rust\library\core\src\option.rs:846:18
|
846 | Some(x) => Some(f(x)),
| ^ trying to reborrow for Unique at alloc84055,
| but parent tag <209678> does not have an
| appropriate item in the borrow stack
|
= help: this indicates a potential bug in the program:
it performed an invalid operation, but the rules it
violated are still experimental
= help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md
for further information
= note: inside `std::option::Option::<std::boxed::Box<fifth::Node<i32>>>::map::<i32, [closure@src\fifth.rs:31:30: 40:10]>` at \lib\rustlib\src\rust\library\core\src\option.rs:846:18
note: inside `fifth::List::<i32>::pop` at src\fifth.rs:31:9
--> src\fifth.rs:31:9
|
31 | / self.head.take().map(|head| {
32 | | let head = *head;
33 | | self.head = head.next;
34 | |
... |
39 | | head.elem
40 | | })
| |__________^
note: inside `fifth::test::basics` at src\fifth.rs:74:20
--> src\fifth.rs:74:20
|
74 | assert_eq!(list.pop(), Some(1));
| ^^^^^^^^^^
note: inside closure at src\fifth.rs:62:5
--> src\fifth.rs:62:5
|
61 | #[test]
| ------- in this procedural macro expansion
62 | / fn basics() {
63 | | let mut list = List::new();
64 | |
65 | | // Check empty list behaves right
... |
96 | | assert_eq!(list.pop(), None);
97 | | }
| |_____^
...
error: aborting due to previous error
Woah. That's one heck of an error.
"Yeah, look at that shit. You love to see it."
Thank you?
"Here take the bottle of estradiol too, you're gonna need it later."
Wait why?
"You're about to think about memory models, trust me."
NARRATOR: The mysterious person then proceeded to transform into a fox and scampered through a hole in the wall. The author then stared into the middle distance for several minutes while they tried to process everything that just happened.
The mysterious fox in the alleyway was right about more than just my gender: miri really is The Good Shit.
Ok so what is miri?
An experimental interpreter for Rust's mid-level intermediate representation (MIR). It can run binaries and test suites of cargo projects and detect certain classes of undefined behavior, for example:
- Out-of-bounds memory accesses and use-after-free
- Invalid use of uninitialized data
- Violation of intrinsic preconditions (an unreachable_unchecked being reached, calling copy_nonoverlapping with overlapping ranges, ...)
- Not sufficiently aligned memory accesses and references
- Violation of some basic type invariants (a bool that is not 0 or 1, for example, or an invalid enum discriminant)
- Experimental: Violations of the Stacked Borrows rules governing aliasing for reference types
- Experimental: Data races (but no weak memory effects)
On top of that, Miri will also tell you about memory leaks: when there is memory still allocated at the end of the execution, and that memory is not reachable from a global static, Miri will raise an error.
...
However, be aware that Miri will not catch all cases of undefined behavior in your program, and cannot run all programs
TL;DR: it interprets your program and notices if you break the rules at runtime and Do An Undefined Behaviour. This is necessary because Undefined Behaviour is generally a thing that happens at runtime. If the issue could be found at compile time, the compiler would just make it an error!
If you're familiar with tools like ubsan and tsan: it's basically that but all together and more extreme.
Miri is now hanging outside the classroom window with a knife. A learning knife.
If we ever want miri to check our work, we can ask them to interpret our test suite with
> cargo +nightly-2022-01-21 miri test
Now let's take a closer look at what they carved into our desk:
error: Undefined Behavior: trying to reborrow for Unique at alloc84055, but parent tag <209678> does not have an appropriate item in the borrow stack
--> \lib\rustlib\src\rust\library\core\src\option.rs:846:18
|
846 | Some(x) => Some(f(x)),
| ^ trying to reborrow for Unique at alloc84055,
| but parent tag <209678> does not have an
| appropriate item in the borrow stack
|
= help: this indicates a potential bug in the program: it
performed an invalid operation, but the rules it
violated are still experimental
= help: see
https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md
for further information
Well I can see we made an error, but that's a confusing error message. What's the "borrow stack"?
We'll try to figure that out in the next section.
Attempting To Understand Stacked Borrows
In the previous section we tried running our unsafe singly-linked queue under miri, and it said we had broken the rules of stacked borrows, and linked us some documentation.
Normally I'd give a guided tour of the docs, but we're not really the target audience of that documentation. It's more designed for compiler developers and academics who are working on the semantics of Rust.
So I'm going to just give you the high level idea of "stacked borrows", and then give you a simple strategy for following the rules.
NARRATOR: Stacked borrows are still "experimental" as a semantic model for Rust, so breaking these rules may not actually mean your program is "wrong". But unless you literally work on the compiler, you should just fix your program when miri complains. Better safe than sorry when it comes to Undefined Behaviour.
The Motivation: Pointer Aliasing
Before we get into what rules we've broken, it will help to understand why the rules exist in the first place. There are a few different motivating problems, but I think the most important one is pointer aliasing.
We say two pointers alias when the pieces of memory they point to overlap. Just as someone who "goes by an alias" can be referred to by two different names, that overlapping piece of memory can be referred to by two different pointers. This can lead to problems.
The compiler uses information about pointer aliasing to optimize accesses to memory, so if the information it has is wrong then the program will be miscompiled and do random garbage.
NARRATOR: Practically speaking, aliasing is more concerned with memory accesses than the pointers themselves, and only really matters when one of the accesses is mutating. Pointers are emphasized because they're a convenient thing to attach rules to.
To understand why pointer aliasing information is important, let's consider The Parable of the Tiny Angry Man.
Michiel was looking through their bookshelf one day when they saw a book they didn't remember. They pulled it from the bookcase and looked at the cover.
"Oh yes, my old copy of War and Peace, a book I definitely have read. I loved the part with all the Peace."
Suddenly there was a knock at the door. Michiel returned the book to its shelf and opened the door -- it was their sworn nemesis Hamslaw. As Hamslaw prepared a devastating remark about Michiel's clearly inferior codegolfing skills, they sensed an opening:
"Hey Hamslaw, have you ever read War and Peace?"
"Pfft, no one's actually read War and Peace."
"Well I have, look it's right there in my bookcase, which obviously means I've read it."
Hamslaw couldn't believe it. Her face shifted from its usual smug demeanor to an iron mask of rage and determination. Hamslaw pushed Michiel aside and power-walked to the book shelf, cleaving the tome from its resting place with the fury of a thousand Valkyries. She turned the ancient text over in her hands, and the instant she saw the cover she began to shake.
Michiel prepared to boast of their clearly unparalleled brilliance, but was interrupted by the sudden laughter of Hamslaw.
"This isn't War and Peace, this is War and Feet!"
Tears were rolling down Hamslaw's face. This was clearly the greatest moment of her life.
"N-no! I just looked at it!"
They grabbed the book from Hamslaw and checked the cover. Indeed, the word "Peace" had been scratched out and replaced with "Feet". Michiel was mortified. This was clearly the worst moment of their life.
They fell to their knees and stared blankly at the bookcase. How could this have happened? They had checked the cover only a moment ago!
And then they saw a bit of motion in the bookcase. It was a tiny man. A tiny many with the angriest scowl Michiel had ever seen. The tiny man flipped Michiel off and mouthed the words "no one will believe you" and disappeared back between the books.
Michiel's plan had been perfect, but they had failed to account for the possibility of a tiny angry man with a sharpie and the desire for destruction. They thought they knew what the cover of the book said, and they thought that no one could have possibly changed it. But alas, they were wrong.
Hamslaw was already working on a zine commemorating her incredible victory — Michiel's reputation at the local Internet Cafe would never recover.
No one wants to be like Michiel, but no one wants to live in constant fear of the tiny angry man either. We want to know when the tiny angry man could be playing tricks on us. When he is, we will be very careful and paranoid about checking everything before we use it. But when the tiny angry man is gone, we want to be able to remember things.
That's the (very simplified) crux of pointer aliasing: when can the compiler assume it's safe to "remember" (cache) values instead of loading them over and over? To know that, the compiler needs to know whenever there could be little angry men mutating the memory behind your back.
NARRATOR: the compiler also uses this information to cache stores, which just means it can avoid committing things to memory if it thinks no one will notice. In this case the problem is still tiny angry men, but they only need to read the memory for it to be a problem.
Safe Stacked Borrows
Ok so we want the compiler to have good pointer aliasing information, can we do that? Well, seemingly Rust is designed for it. Mutable references aren't aliased by definition, and although shared references can alias eachother, they can't mutate. Perfect! Ship it!
Except it's more complicated than that. We can "reborrow" mutable pointers like this:
#![allow(unused)] fn main() { let mut data = 10; let ref1 = &mut data; let ref2 = &mut *ref1; *ref2 += 2; *ref1 += 1; println!("{}", data); }
The compiles and runs fine. What's the deal?
Well we can see what's going on by swapping the two uses:
let mut data = 10;
let ref1 = &mut data;
let ref2 = &mut *ref1;
// ORDER SWAPPED!
*ref1 += 1;
*ref2 += 2;
println!("{}", data);
error[E0503]: cannot use `*ref1` because it was mutably borrowed
--> src/main.rs:6:5
|
4 | let ref2 = &mut *ref1;
| ---------- borrow of `*ref1` occurs here
5 |
6 | *ref1 += 1;
| ^^^^^^^^^^ use of borrowed `*ref1`
7 | *ref2 += 2;
| ---------- borrow later used here
For more information about this error, try `rustc --explain E0503`.
error: could not compile `playground` due to previous error
It's suddenly a compiler error!
When we reborrow a mutable pointer, the original pointer can't be used anymore until the borrower is done with it (no more uses).
In the code that works, there's a nice little nesting of the uses. We reborrow the pointer, use the new pointer for a while, and then stop using it before using the older pointer again. In the code that doesn't work, that doesn't happen. We just interleave the uses arbitrarily.
This is how we can have reborrows and still have aliasing information: all of our reborrows clearly nest, so we can consider only one of them "live" at any given time.
Hey, you know what's a great way to represent cleanly nested things? A stack. A stack of borrows.
Oh hey it's Stacked Borrows!
Whatever's at the top of the borrow stack is "live" and knows it's effectively unaliased. When you reborrow a pointer, the new pointer is pushed onto the stack, becoming the live pointer. When you use an older pointer it's brought back to life by popping everything on the borrow stack above it. At this point the pointer "knows" it was reborrowed and that the memory might have been modified, but that it once more has exclusive access -- no need to worry about little angry men.
So it's actually always ok to access a reborrowed pointer, because we can always pop everything above it. The real trouble is accessing a pointer that has already been popped off of the borrow stack -- then you've messed up.
Thankfully the design of the borrowchecker ensures that safe Rust programs follow these rules, as we saw in the above example, but the compiler generally views this problem "backwards" from the stacked borrows perspective. Instead of saying using ref1 invalidates ref2, it insists that ref2 must be valid for all its uses, and that ref1 is the one messing things up by going out of turn.
Hence "cannot use *ref1 because it was mutably borrowed". It's the same result (especially with non-lexical lifetimes), but framed in a way that's probably more intuitive.
But the borrowchecker can't help us when we start using unsafe pointers!
Unsafe Stacked Borrows
So we want to somehow have a way for unsafe pointers to participate in this stacked borrows system, even though the compiler can't track them properly. And we also want the system to be fairly permissive so that it's not too easy to mess it up and cause UB.
That's a hard problem, and I don't know how to solve it, but the folks who worked on Stacked Borrows came up with something plausible, and miri tries to implement it.
The very high-level concept is that when you convert a reference (or any other safe pointer) into an raw pointer it's basically like taking a reborrow. So now the raw pointer is allowed to do whatever it wants with that memory, and when the reborrow expires it's just like when that happens with normal reborrows.
But the question is, when does that reborrow expire? Well, probably a good time to expire it is when you start using the original reference again. Otherwise things aren't a nice nested stack.
But wait, you can turn a raw pointer into a reference! And you can copy raw pointers! What if you go &mut -> *mut -> &mut -> *mut and then access the first *mut? How the heck do the stacked borrows work then?
I genuinely don't know! That's why things are complicated. In fact they're extra complicated because stacked borrows are trying to be more permissive and let more unsafe code work the way you'd expect it to. This is why I run things under miri to try to help me catch mistakes.
In fact, this messiness is why there is an extra-experimental extra-strict mode of miri: -Zmiri-tag-raw-pointers.
To enable it, we need to pass it via a MIRIFLAGS environment variable like this:
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri test
Or like this on Windows, where you need to just set the variable globally:
$env:MIRIFLAGS="-Zmiri-tag-raw-pointers"
cargo +nightly-2022-01-21 miri test
We'll generally be trying to conform to this extra-strict mode just to be extra confident in our work. It's also in some sense "simpler", so it's actually better for messing around and getting an intuition for stacked borrows.
Managing Stacked Borrows
So when using raw pointers we're going to try to stick to a heuristic that's simple and blunt and will hopefully have a large margin of error:
Once you start using raw pointers, try to ONLY use raw pointers.
This makes it as unlikely as possible to accidentally lose the raw pointer's "permission" to access the memory.
NARRATOR: this is oversimplified in two regards:
Safe pointers often assert more properties than just aliasing: the memory is allocated, it's aligned, it's large enough to fit the type of the pointee, the pointee is properly initialized, etc. So it's even more dangerous to wildly throw them around when things are in a dubious state.
Even if you stay in raw pointer land, you can't just wildly alias any memory. Pointers are conceptually tied to specific "allocations" (which can be as granular as a local variable on the stack), and you're not supposed to take a pointer from one allocation, offset it, and then access memory in a different allocation. If this was allowed, there would always be the threat of tiny angry men everywhere. This is part of the reason "pointers are just integers" is a problematic viewpoint.
Now, we still want safe references in our interface, because we want to build a nice safe abstraction so the user of our list doesn't have to know or worry about.
So what we're going to do is:
- At the start of a method, use the input references to get our raw pointers
- Do our best to only use unsafe pointers from this point on
- Convert our raw pointers back to safe pointers at the end if needed
But the fields of our types are private so we're going to keep those entirely as raw pointers.
In fact, part of the big mistake we made was continuing to use Box! Box has a special annotation in it that tells the compiler "hey this is a lot like &mut, because it uniquely owns that pointer". Which is true!
But the raw pointer we were keeping to the end of the list was pointing into a Box, so whenever we access the Box normally we're probably invalidating that raw pointer's "reborrow"! ☠
In the next section we'll return to our true form and hit our heads against a bunch of examples.
Testing Stacked Borrows
TL;DR of the previous section's (simplified) memory model for Rust:
- Rust conceptually handles reborrows by maintaining a "borrow stack"
- Only the one on the top of the stack is "live" (has exclusive access)
- When you access a lower one it becomes "live" and the ones above it get popped
- You're not allowed to use pointers that have been popped from the borrow stack
- The borrowchecker ensures safe code code obeys this
- Miri theoretically checks that raw pointers obey this at runtime
That was a lot of theory and ideas -- let's move on to the true heart and soul of this book: writing some bad code and getting our tools to scream at us. We're going to go through a ton of examples to try to see if our mental model makes sense, and to try to get an intuitive feel for stacked borrows.
NARRATOR: Catching Undefined Behaviour in practice is a hairy business. After all, you're dealing with situations that the compiler literally assumes don't happen.
If you're lucky, things will "seem to work" today, but they'll be a ticking time bomb for a Smarter Compiler or slight change to the code. If you're really lucky things will reliably crash so you can just catch the mistake and fix it. But if you're unlucky, then things will be broken in weird and baffling ways.
Miri tries to work around this by getting rustc's most naive and unoptimized view of the program and tracking extra state as it interprets. As far as "sanitizers" go, this is a fairly deterministic and robust approach but it will never be perfect. You need your test program to actually have an execution with that UB, and for a large enough program it's very easy to introduce all sorts of non-determinism (HashMaps use RNG by default!).
We can never take miri approving of our program's execution as an absolute certain statement there's no UB. It's also possible for miri to think something's UB when it really isn't. But if we have a mental model of how things work, and miri seems to agree with us, that's a good sign that we're on the right track.
Basic Borrows
In previous sections we saw that the borrowchecker didn't like this code:
let mut data = 10;
let ref1 = &mut data;
let ref2 = &mut *ref1;
// ORDER SWAPPED!
*ref1 += 1;
*ref2 += 2;
println!("{}", data);
Let's see what happens when we replace ref2 with *mut:
unsafe {
let mut data = 10;
let ref1 = &mut data;
let ptr2 = ref1 as *mut _;
// ORDER SWAPPED!
*ref1 += 1;
*ptr2 += 2;
println!("{}", data);
}
cargo run
Compiling miri-sandbox v0.1.0
Finished dev [unoptimized + debuginfo] target(s) in 0.71s
Running `target\debug\miri-sandbox.exe`
13
Rustc seems perfectly happy with this: no warnings and the program produced the result we expected! Now let's look at what miri (in strict mode) thinks of it:
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
Finished dev [unoptimized + debuginfo] target(s) in 0.00s
Running cargo-miri.exe target\miri
error: Undefined Behavior: no item granting read access
to tag <untagged> at alloc748 found in borrow stack.
--> src\main.rs:9:9
|
9 | *ptr2 += 2;
| ^^^^^^^^^^ no item granting read access to tag <untagged>
| at alloc748 found in borrow stack.
|
= help: this indicates a potential bug in the program:
it performed an invalid operation, but the rules it
violated are still experimental
Nice! Our intuitive model of how things work held up: although the compiler couldn't catch the issue for us, miri did.
Let's try something more complicated, the &mut -> *mut -> &mut -> *mut case we alluded to before:
unsafe {
let mut data = 10;
let ref1 = &mut data;
let ptr2 = ref1 as *mut _;
let ref3 = &mut *ptr2;
let ptr4 = ref3 as *mut _;
// Access the first raw pointer first
*ptr2 += 2;
// Then access things in "borrow stack" order
*ptr4 += 4;
*ref3 += 3;
*ptr2 += 2;
*ref1 += 1;
println!("{}", data);
}
cargo run
22
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
error: Undefined Behavior: no item granting read access
to tag <1621> at alloc748 found in borrow stack.
--> src\main.rs:13:5
|
13 | *ptr4 += 4;
| ^^^^^^^^^^ no item granting read access to tag <1621>
| at alloc748 found in borrow stack.
|
Wow yep! In strict mode miri can "tell apart" the two raw pointers and have using the second one invalidate the first one. Let's see if everything works when we remove the first use that messes everything up:
unsafe {
let mut data = 10;
let ref1 = &mut data;
let ptr2 = ref1 as *mut _;
let ref3 = &mut *ptr2;
let ptr4 = ref3 as *mut _;
// Access things in "borrow stack" order
*ptr4 += 4;
*ref3 += 3;
*ptr2 += 2;
*ref1 += 1;
println!("{}", data);
}
cargo run
20
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
20
NICE.
Yeah I'm pretty sure at this point we can all get PhD's in programming language memory model design and implementation. Who even needs compilers, this stuff is easy.
NARRATOR: it was not, but I'm proud of you nonetheless.
Testing Arrays
Let's mess with some arrays and pointer offsets (add and sub). This should work, right?
unsafe {
let mut data = [0; 10];
let ref1_at_0 = &mut data[0]; // Reference to 0th element
let ptr2_at_0 = ref1_at_0 as *mut i32; // Ptr to 0th element
let ptr3_at_1 = ptr2_at_0.add(1); // Ptr to 1st element
*ptr3_at_1 += 3;
*ptr2_at_0 += 2;
*ref1_at_0 += 1;
// Should be [3, 3, 0, ...]
println!("{:?}", &data[..]);
}
cargo run
[3, 3, 0, 0, 0, 0, 0, 0, 0, 0]
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
error: Undefined Behavior: no item granting read access
to tag <1619> at alloc748+0x4 found in borrow stack.
--> src\main.rs:8:5
|
8 | *ptr3_at_1 += 3;
| ^^^^^^^^^^^^^^^ no item granting read access to tag <1619>
| at alloc748+0x4 found in borrow stack.
Rips up gradschool application
What happened? We're using the borrow stack perfectly fine! Does something weird happen when we go ptr -> ptr? What if we just copy the pointer so they all go to the same location:
#![allow(unused)] fn main() { unsafe { let mut data = [0; 10]; let ref1_at_0 = &mut data[0]; // Reference to 0th element let ptr2_at_0 = ref1_at_0 as *mut i32; // Ptr to 0th element let ptr3_at_0 = ptr2_at_0; // Ptr to 0th element *ptr3_at_0 += 3; *ptr2_at_0 += 2; *ref1_at_0 += 1; // Should be [6, 0, 0, ...] println!("{:?}", &data[..]); } }
cargo run
[6, 0, 0, 0, 0, 0, 0, 0, 0, 0]
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
[6, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Nope, that works fine. Maybe we're getting lucky, let's just make a real big mess of pointers:
#![allow(unused)] fn main() { unsafe { let mut data = [0; 10]; let ref1_at_0 = &mut data[0]; // Reference to 0th element let ptr2_at_0 = ref1_at_0 as *mut i32; // Ptr to 0th element let ptr3_at_0 = ptr2_at_0; // Ptr to 0th element let ptr4_at_0 = ptr2_at_0.add(0); // Ptr to 0th element let ptr5_at_0 = ptr3_at_0.add(1).sub(1); // Ptr to 0th element // An absolute jumbled hash of ptr usages *ptr3_at_0 += 3; *ptr2_at_0 += 2; *ptr4_at_0 += 4; *ptr5_at_0 += 5; *ptr3_at_0 += 3; *ptr2_at_0 += 2; *ref1_at_0 += 1; // Should be [20, 0, 0, ...] println!("{:?}", &data[..]); } }
cargo run
[20, 0, 0, 0, 0, 0, 0, 0, 0, 0]
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
[20, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Nope! Miri is actually way more permissive when it comes to raw pointers that are derived from other raw pointers. They all share the same "borrow" (or miri calls it, a tag).
Once you start using raw pointers they can freely split into their own tiny angry men and mess with themselves. This is ok because the compiler understands that and won't optimize the reads and writes the same it does with references.
NARRATOR: If the code is simple enough, the compiler can keep track of all the derived pointers and still optimize things where possible, but it's going to be a lot more brittle than the reasoning it can use for references.
So what's the real problem?
Even though data is one "allocation" (local variable), ref1_at_0 is only borrowing the first element. Rust allows borrows to be broken up so that they only apply to particular parts of the allocation! Let's try it out:
unsafe {
let mut data = [0; 10];
let ref1_at_0 = &mut data[0]; // Reference to 0th element
let ref2_at_1 = &mut data[1]; // Reference to 1th element
let ptr3_at_0 = ref1_at_0 as *mut i32; // Ptr to 0th element
let ptr4_at_1 = ref2_at_1 as *mut i32; // Ptr to 1th element
*ptr4_at_1 += 4;
*ptr3_at_0 += 3;
*ref2_at_1 += 2;
*ref1_at_0 += 1;
// Should be [3, 3, 0, ...]
println!("{:?}", &data[..]);
}
error[E0499]: cannot borrow `data[_]` as mutable more than once at a time
--> src\main.rs:5:21
|
4 | let ref1_at_0 = &mut data[0]; // Reference to 0th element
| ------------ first mutable borrow occurs here
5 | let ref2_at_1 = &mut data[1]; // Reference to 1th element
| ^^^^^^^^^^^^ second mutable borrow occurs here
6 | let ptr3_at_0 = ref1_at_0 as *mut i32; // Ptr to 0th element
| --------- first borrow later used here
|
= help: consider using `.split_at_mut(position)` or similar method
to obtain two mutable non-overlapping sub-slices
Shoot! Rust doesn't track array indices to prove these borrows are disjoint, but it does give us split_at_mut to break a slice into multiple parts in a way that is safe to assume works:
#![allow(unused)] fn main() { unsafe { let mut data = [0; 10]; let slice1 = &mut data[..]; let (slice2_at_0, slice3_at_1) = slice1.split_at_mut(1); let ref4_at_0 = &mut slice2_at_0[0]; // Reference to 0th element let ref5_at_1 = &mut slice3_at_1[0]; // Reference to 1th element let ptr6_at_0 = ref4_at_0 as *mut i32; // Ptr to 0th element let ptr7_at_1 = ref5_at_1 as *mut i32; // Ptr to 1th element *ptr7_at_1 += 7; *ptr6_at_0 += 6; *ref5_at_1 += 5; *ref4_at_0 += 4; // Should be [10, 12, 0, ...] println!("{:?}", &data[..]); } }
cargo run
[10, 12, 0, 0, 0, 0, 0, 0, 0, 0]
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
[10, 12, 0, 0, 0, 0, 0, 0, 0, 0]
Hey, that works! Slices properly tell the compiler and miri "hey I'm taking a huge loan on all of the memory in my range", so they know all of the elements can be mutated.
Also note that operations like split_at_mut being allowed tells us that borrows can be less of a stack and more of a tree, because we can break one big borrow into a bunch of disjoint smaller ones, and everything still works.
(I think in the actual stacked borrows model everything's still stacks because the stacks are conceptually tracking permissions for each byte of the program..?)
What if we directly turn a slice into a pointer? Will that pointer have access to the full slice?
#![allow(unused)] fn main() { unsafe { let mut data = [0; 10]; let slice1_all = &mut data[..]; // Slice for the entire array let ptr2_all = slice1_all.as_mut_ptr(); // Pointer for the entire array let ptr3_at_0 = ptr2_all; // Pointer to 0th elem (the same) let ptr4_at_1 = ptr2_all.add(1); // Pointer to 1th elem let ref5_at_0 = &mut *ptr3_at_0; // Reference to 0th elem let ref6_at_1 = &mut *ptr4_at_1; // Reference to 1th elem *ref6_at_1 += 6; *ref5_at_0 += 5; *ptr4_at_1 += 4; *ptr3_at_0 += 3; // Just for fun, modify all the elements in a loop // (Could use any of the raw pointers for this, they share a borrow!) for idx in 0..10 { *ptr2_all.add(idx) += idx; } // Safe version of this same code for fun for (idx, elem_ref) in slice1_all.iter_mut().enumerate() { *elem_ref += idx; } // Should be [8, 12, 4, 6, 8, 10, 12, 14, 16, 18] println!("{:?}", &data[..]); } }
cargo run
[8, 12, 4, 6, 8, 10, 12, 14, 16, 18]
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
[8, 12, 4, 6, 8, 10, 12, 14, 16, 18]
Nice! Pointers aren't just integers: they have a range of memory associated with them, and with Rust we're allowed to narrow that range!
Testing Shared References
In all of these examples I have been very carefully only using mutable references and doing read-modify-write operations (+=) to keep things as simple as possible.
But Rust has shared references that are read-only and can be freely copied, how should those work? Well we've seen that raw pointers can be freely copied and we can handle that by saying they "share" a single borrow. Maybe we think of shared references the same way?
Let's test that out with a function that reads a value (println! can be a little magical with auto-ref/deref stuff, so I'm wrapping it in a function to make sure we're testing what we want to be):
fn opaque_read(val: &i32) {
println!("{}", val);
}
unsafe {
let mut data = 10;
let mref1 = &mut data;
let sref2 = &mref1;
let sref3 = sref2;
let sref4 = &*sref2;
// Random hash of shared reference reads
opaque_read(sref3);
opaque_read(sref2);
opaque_read(sref4);
opaque_read(sref2);
opaque_read(sref3);
*mref1 += 1;
opaque_read(&data);
}
cargo run
warning: unnecessary `unsafe` block
--> src\main.rs:6:1
|
6 | unsafe {
| ^^^^^^ unnecessary `unsafe` block
|
= note: `#[warn(unused_unsafe)]` on by default
warning: `miri-sandbox` (bin "miri-sandbox") generated 1 warning
10
10
10
10
10
11
Oh yeah we forgot to do anything with raw pointers, but at least we can see that it's fine for all the shared references to be used interchangeably. Now let's mix in some raw pointers:
fn opaque_read(val: &i32) {
println!("{}", val);
}
unsafe {
let mut data = 10;
let mref1 = &mut data;
let ptr2 = mref1 as *mut i32;
let sref3 = &mref1;
let ptr4 = sref3 as *mut i32;
*ptr4 += 4;
opaque_read(sref3);
*ptr2 += 2;
*mref1 += 1;
opaque_read(&data);
}
cargo run
error[E0606]: casting `&&mut i32` as `*mut i32` is invalid
--> src\main.rs:11:16
|
11 | let ptr4 = sref3 as *mut i32;
| ^^^^^^^^^^^^^^^^^
Oh whoops, we were actually messing around with & &mut instead of &! Rust is very good at papering over that when it doesn't matter. Let's properly reborrow it with let sref3 = &*mref1:
cargo run
error[E0606]: casting `&i32` as `*mut i32` is invalid
--> src\main.rs:11:16
|
11 | let ptr4 = sref3 as *mut i32;
| ^^^^^^^^^^^^^^^^^
Nope, Rust still doesn't like that! You can only cast a shared reference to a *const which can only read. But what if we just... do... this...?
let ptr4 = sref3 as *const i32 as *mut i32;
cargo run
14
17
WHAT. OK SURE FINE? Great cast system there Rust. It's almost like the *const is a pretty useless type that only really exists to describe C APIs and to vaguely suggest correct usage (it is, it does). What does miri think?
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
error: Undefined Behavior: no item granting write access to
tag <1621> at alloc742 found in borrow stack.
--> src\main.rs:13:5
|
13 | *ptr4 += 4;
| ^^^^^^^^^^ no item granting write access to tag <1621>
| at alloc742 found in borrow stack.
Alas, though we can get around the compiler complaining with a double cast, it doesn't actually make this operation allowed. When we take the shared reference, we're promising not to modify the value.
This is important because that means when the shared borrow is popped off the borrow stack, the mutable pointers below it can assume the memory hasn't changed. There may have been some tiny angry men reading the memory (so writes had to be comitted) but they weren't able to modify it and the mutable pointers can assume the last value they wrote is still there!
Once a shared reference is on the borrow-stack, everything that gets pushed on top of it only has read permissions.
We can however do this:
#![allow(unused)] fn main() { fn opaque_read(val: &i32) { println!("{}", val); } unsafe { let mut data = 10; let mref1 = &mut data; let ptr2 = mref1 as *mut i32; let sref3 = &*mref1; let ptr4 = sref3 as *const i32 as *mut i32; opaque_read(&*ptr4); opaque_read(sref3); *ptr2 += 2; *mref1 += 1; opaque_read(&data); } }
Note how it was still "fine" to create a mutable raw pointer as long as we only actually read from it!
cargo run
10
10
13
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
10
10
13
And just to be sure, let's check that a shared reference gets popped like normal:
fn opaque_read(val: &i32) {
println!("{}", val);
}
unsafe {
let mut data = 10;
let mref1 = &mut data;
let ptr2 = mref1 as *mut i32;
let sref3 = &*mref1;
*ptr2 += 2;
opaque_read(sref3); // Read in the wrong order?
*mref1 += 1;
opaque_read(&data);
}
cargo run
12
13
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
error: Undefined Behavior: trying to reborrow for SharedReadOnly
at alloc742, but parent tag <1620> does not have an appropriate
item in the borrow stack
--> src\main.rs:13:17
|
13 | opaque_read(sref3); // Read in the wrong order?
| ^^^^^ trying to reborrow for SharedReadOnly
| at alloc742, but parent tag <1620>
| does not have an appropriate item
| in the borrow stack
|
Hey, we even got a slightly different error message about SharedReadOnly instead of some specific tag. That makes sense: once there's any shared references, basically everything else is just a big SharedReadOnly soup so there's no need to distinguish any of them!
Testing Interior Mutability
Remember that really horrible chapter of the book where we tried to make a linked list with RefCell and Rc and everything was even worse than usual when trying to write this godforsaken linked lists?
We've been insisting shared references can't be used for mutation but that chapter was all about how you could actually mutate through shared references with interior mutability. Let's try the nice and simple std::cell::Cell type:
#![allow(unused)] fn main() { use std::cell::Cell; unsafe { let mut data = Cell::new(10); let mref1 = &mut data; let ptr2 = mref1 as *mut Cell<i32>; let sref3 = &*mref1; sref3.set(sref3.get() + 3); (*ptr2).set((*ptr2).get() + 2); mref1.set(mref1.get() + 1); println!("{}", data.get()); } }
Ah, such a beautiful mess. It will be lovely to see miri spit on it.
cargo run
16
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
16
Wait, really? That's fine? Why? How? What even is a Cell?
Smashes the padlock on the stdlib
pub struct Cell<T: ?Sized> {
value: UnsafeCell<T>,
}
What the heck is UnsafeCell?
Smashes another padlock just to really show the stdlib we mean business
#[lang = "unsafe_cell"]
#[repr(transparent)]
#[repr(no_niche)]
pub struct UnsafeCell<T: ?Sized> {
value: T,
}
Oh it's wizard magic. Ok. I guess. #[lang = "unsafe_cell"] is literally just saying UnsafeCell is UnsafeCell. Let's stop breaking locks and check the actual documentation of std::cell::UnsafeCell.
The core primitive for interior mutability in Rust.
If you have a reference
&T, then normally in Rust the compiler performs optimizations based on the knowledge that&Tpoints to immutable data. Mutating that data, for example through an alias or by transmuting an&Tinto an&mut T, is considered undefined behavior.UnsafeCell<T>opts-out of the immutability guarantee for&T: a shared reference&UnsafeCell<T>may point to data that is being mutated. This is called “interior mutability”.
Oh it really is just wizard magic.
UnsafeCell basically tells the compiler "hey listen, we're gonna get goofy with this memory, don't make any of the usual aliasing assumptions about it". Like putting up a big "CAUTION: TINY ANGRY MEN CROSSING" sign.
Let's see how adding UnsafeCell makes miri happy:
use std::cell::UnsafeCell;
fn opaque_read(val: &i32) {
println!("{}", val);
}
unsafe {
let mut data = UnsafeCell::new(10);
let mref1 = data.get_mut(); // Get a mutable ref to the contents
let ptr2 = mref1 as *mut i32;
let sref3 = &*ptr2;
*ptr2 += 2;
opaque_read(sref3);
*mref1 += 1;
println!("{}", *data.get());
}
cargo run
12
13
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
error: Undefined Behavior: trying to reborrow for SharedReadOnly
at alloc748, but parent tag <1629> does not have an appropriate
item in the borrow stack
--> src\main.rs:15:17
|
15 | opaque_read(sref3);
| ^^^^^ trying to reborrow for SharedReadOnly
| at alloc748, but parent tag <1629> does
| not have an appropriate item in the
| borrow stack
|
Wait, what? We spoke the magic words! What am I going to do with all this federally approved ritual-enhancing goat blood?
Well, we did, but then we completely discarded the spell by using get_mut which peeks inside the UnsafeCell and makes a proper &mut i32 to it anyway!
Think about it: if the compiler had to assume &mut i32 could be looking inside an UnsafeCell, then it would never be able to make any assumptions about aliasing at all! Everything could be full of tiny angry men.
So what we need to do is keep the UnsafeCell in our pointer types so that the compiler understands what we're doing.
#![allow(unused)] fn main() { use std::cell::UnsafeCell; fn opaque_read(val: &i32) { println!("{}", val); } unsafe { let mut data = UnsafeCell::new(10); let mref1 = &mut data; // Mutable ref to the *outside* let ptr2 = mref1.get(); // Get a raw pointer to the insides let sref3 = &*mref1; // Get a shared ref to the *outside* *ptr2 += 2; // Mutate with the raw pointer opaque_read(&*sref3.get()); // Read from the shared ref *sref3.get() += 3; // Write through the shared ref *mref1.get() += 1; // Mutate with the mutable ref println!("{}", *data.get()); } }
cargo run
12
16
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
12
16
It works! I won't have to throw out all this blood after all.
Actually, hey wait. We're still being a bit goofy with the order here. We made ptr2 first, and then made sref3 from the mutable pointer. And then we used the raw pointer before the shared pointer. That all seems... wrong.
Actually wait we did that with the Cell example too. HMMM.
We're forced to conclude one of two things:
- Miri is imperfect and this is actually still UB.
- Our simplified model is in fact an oversimplication.
I'd put my money on the second one, but just to be safe let's make a version that's definitely airtight in our simplified model of stacked borrows:
#![allow(unused)] fn main() { use std::cell::UnsafeCell; fn opaque_read(val: &i32) { println!("{}", val); } unsafe { let mut data = UnsafeCell::new(10); let mref1 = &mut data; // These two are swapped so the borrows are *definitely* totally stacked let sref2 = &*mref1; // Derive the ptr from the shared ref to be super safe! let ptr3 = sref2.get(); *ptr3 += 3; opaque_read(&*sref2.get()); *sref2.get() += 2; *mref1.get() += 1; println!("{}", *data.get()); } }
cargo run
13
16
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
13
16
Now, one reason why the first implementation we had might actually be correct is because if you really think about it &UnsafeCell<T> really is no different from *mut T as far as aliasing is concerned. You can infinitely copy it and mutate through it!
So in some sense we just created two raw pointers and used them interchangeably like normal. It's a little sketchy that both were derived from the mutable reference, so maybe the second one's creation should still pop the first one off the borrow stack, but that's not really necessary since we're not actually accessing the contents of the mutable reference, just copying its address.
A line like let sref2 = &*mref1 is a tricksy thing. Syntactically it looks like we're dereferencing it, but dereferencing on it's own isn't actually a thing? Consider &my_tuple.0: you aren't actually doing anything to my_tuple or .0, you're just using them to refer to a location in memory and putting & in front of it that says "don't load this, just write the address down".
&* is the same thing: the * is just saying "hey let's talk about the location this pointer points to" and the & is just saying "now write that address down". Which is of course the same value the original pointer had. But the type of the pointer has changed, because, uh, types!
That said, if you do &** then you are in fact loading a value with the first *! * is weird!
NARRATOR: No one cares that you know the word "lvalue", Jonathan. In Rust we call them places, which is totally different and so much cooler?
Testing Box
Hey remember why we started this extremely long aside? You don't? Weird.
Well it was because we mixed Box and raw pointers. Box is kind of like &mut, because it claims unique ownership of the memory it points to. Let's test that claim out:
unsafe {
let mut data = Box::new(10);
let ptr1 = (&mut *data) as *mut i32;
*data += 10;
*ptr1 += 1;
// Should be 21
println!("{}", data);
}
cargo run
21
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
error: Undefined Behavior: no item granting read access
to tag <1707> at alloc763 found in borrow stack.
--> src\main.rs:7:5
|
7 | *ptr1 += 1;
| ^^^^^^^^^^ no item granting read access to tag <1707>
| at alloc763 found in borrow stack.
|
Yep, miri hates that. Let's check that doing things in the right order is ok:
#![allow(unused)] fn main() { unsafe { let mut data = Box::new(10); let ptr1 = (&mut *data) as *mut i32; *ptr1 += 1; *data += 10; // Should be 21 println!("{}", data); } }
cargo run
21
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri run
21
Yep!
Whelp that's all folks, we're finally done talking and thinking about stacked borrows!
...wait how do we solve this problem with Box? Like, sure we can write toy programs like this but we need to store the Box somewhere and hold onto our raw pointers for a potentially long time. Surely stuff will get mixed up and invalidated?
Great question! To answer that we'll finally be returning to our true calling: writing some god damn linked lists.
Wait, I need to write linked lists again? Let's not be hasty folks. Be reasonable. Just hold on I'm sure there's some other interesting issues for me to discu—
Layout and Basics 2: Getting Raw
TL;DR on the previous three sections: randomly mixing safe pointers like
&,&mut, andBoxwith unsafe pointers like*mutand*constis a recipe for Undefined Behaviour because the safe pointers introduce extra constraints that we aren't obeying with the raw pointers.
Oh god I need to write linked lists again. Fine. FINE. It's Fine. We're fine.
We're gonna knock a lot of this section out real quick since we already discussed the design in the first try around, and everything we did was basically correct except for how we mixed together safe and unsafe pointers.
Layout
So in the new layout we're only going to only use raw pointers and everything will be perfect and we'll never make mistakes again.
Here's our old broken layout:
#![allow(unused)] fn main() { pub struct List<T> { head: Link<T>, tail: *mut Node<T>, // INNOCENT AND KIND } type Link<T> = Option<Box<Node<T>>>; // THE REAL EVIL struct Node<T> { elem: T, next: Link<T>, } }
And here's our new layout:
#![allow(unused)] fn main() { pub struct List<T> { head: Link<T>, tail: *mut Node<T>, } type Link<T> = *mut Node<T>; // MUCH BETTER struct Node<T> { elem: T, next: Link<T>, } }
Remember: Option isn't as nice or useful when we're using raw pointers, so we're not using that anymore. In later sections we'll look at the NonNull type, but don't worry about that for now.
Basics
List::new is basically the same.
use ptr;
impl<T> List<T> {
pub fn new() -> Self {
List { head: ptr::null_mut(), tail: ptr::null_mut() }
}
}
Push is basically the s-
pub fn push(&mut self, elem: T) {
let mut new_tail = Box::new(
Wait we're not using Box anymore. How do we allocate memory without Box?
Well, we could with std::alloc::alloc, but that's like bringing a katana into the kitchen. It'll get the job done but it's kinda overkill and unwieldy.
We want to have boxes, but, not. One completely wild but maybe viable option would be to do something like this:
struct Node<T> {
elem: T,
real_next: Option<Box<Node<T>>>,
next: *mut Node<T>,
}
With the idea that we create the Boxes and store them in our node, but then we take a raw pointer into them and only use that raw pointer until we're done with the Node and want to destroy it. Then we can take the Box out of real_next and drop it. I think that would conform to our very simplified stacked borrows model?
If you wanna try to make that, have "fun", but that just looks awful right? This isn't the chapter on Rc and RefCell, we're not gonna play this game anymore. We're gonna just make simple and clean stuff.
So instead we're going to use the very nice Box::into_raw function:
pub fn into_raw(b: Box<T>) -> *mut TConsumes the Box, returning a wrapped raw pointer.
The pointer will be properly aligned and non-null.
After calling this function, the caller is responsible for the memory previously managed by the Box. In particular, the caller should properly destroy T and release the memory, taking into account the memory layout used by Box. The easiest way to do this is to convert the raw pointer back into a Box with the
Box::from_rawfunction, allowing the Box destructor to perform the cleanup.Note: this is an associated function, which means that you have to call it as
Box::into_raw(b)instead ofb.into_raw(). This is so that there is no conflict with a method on the inner type.Examples
Converting the raw pointer back into a Box with Box::from_raw for automatic cleanup:
let x = Box::new(String::from("Hello")); let ptr = Box::into_raw(x); let x = unsafe { Box::from_raw(ptr) };
Nice, that looks literally designed for our use case. It also matches the rules we're trying to follow: start with safe stuff, turn into into raw pointers, and then only convert back to safe stuff at the end (when we want to Drop it).
This is basically exactly like doing the weird real_next thing but without having to faff around storing the Box when it's the exact same pointer as the raw pointer anyway.
Also now that we're just using raw pointers everywhere, let's not worry about keeping those unsafe blocks narrow: it's all unsafe now. (It always was, but it's nice to lie to yourself sometimes.)
pub fn push(&mut self, elem: T) {
unsafe {
// Immediately convert the Box into a raw pointer
let new_tail = Box::into_raw(Box::new(Node {
elem: elem,
next: ptr::null_mut(),
}));
if !self.tail.is_null() {
(*self.tail).next = new_tail;
} else {
self.head = new_tail;
}
self.tail = new_tail;
}
}
Hey that code's actually looking a lot cleaner now that we're sticking to raw pointers!
On to pop, which is also pretty similar to how we left it, although we've got to remember to use Box::from_raw to clean up the allocation:
pub fn pop(&mut self) -> Option<T> {
unsafe {
if self.head.is_null() {
None
} else {
// RISE FROM THE GRAVE
let head = Box::from_raw(self.head);
self.head = head.next;
if self.head.is_null() {
self.tail = ptr::null_mut();
}
Some(head.elem)
}
}
}
Our nice little takes and maps are dead, gotta just check and set null manually now.
And while we're here, let's slap in the destructor. This time we'll implement it as just repeatedly popping, because it's cute and simple:
impl<T> Drop for List<T> {
fn drop(&mut self) {
while let Some(_) = self.pop() { }
}
}
Now, for the moment of truth:
#[cfg(test)]
mod test {
use super::List;
#[test]
fn basics() {
let mut list = List::new();
// Check empty list behaves right
assert_eq!(list.pop(), None);
// Populate list
list.push(1);
list.push(2);
list.push(3);
// Check normal removal
assert_eq!(list.pop(), Some(1));
assert_eq!(list.pop(), Some(2));
// Push some more just to make sure nothing's corrupted
list.push(4);
list.push(5);
// Check normal removal
assert_eq!(list.pop(), Some(3));
assert_eq!(list.pop(), Some(4));
// Check exhaustion
assert_eq!(list.pop(), Some(5));
assert_eq!(list.pop(), None);
// Check the exhaustion case fixed the pointer right
list.push(6);
list.push(7);
// Check normal removal
assert_eq!(list.pop(), Some(6));
assert_eq!(list.pop(), Some(7));
assert_eq!(list.pop(), None);
}
}
cargo test
running 12 tests
test fifth::test::basics ... ok
test first::test::basics ... ok
test fourth::test::basics ... ok
test fourth::test::peek ... ok
test second::test::basics ... ok
test fourth::test::into_iter ... ok
test second::test::into_iter ... ok
test second::test::iter ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test third::test::basics ... ok
test third::test::iter ... ok
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured
Good, but does miri agree?
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri test
running 12 tests
test fifth::test::basics ... ok
test first::test::basics ... ok
test fourth::test::basics ... ok
test fourth::test::peek ... ok
test second::test::basics ... ok
test fourth::test::into_iter ... ok
test second::test::into_iter ... ok
test second::test::iter ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test third::test::basics ... ok
test third::test::iter ... ok
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured
EYYYY!!!!!
IT FRIGGIN WORKED!
PROBABLY!
FAILING TO FIND UNDEFINED BEHAVIOUR IS NOT A PROOF THAT IT ISN'T THERE WAITING TO CAUSE PROBLEMS BUT THERE IS A LIMIT TO HOW RIGOROUS I AM WILLING TO BE FOR A JOKE BOOK ABOUT LINKED LISTS SO WE'RE GONNA CALL THIS A 100% MACHINE VERIFIED PROOF AND ANYONE WHO SAYS OTHERWISE CAN SUCK MY COQ!
∴ QED □
Extra Junk
Now that push and pop are written, everything else is acutally exactly the same as the stack case, weirdly. Only operations that change the length of the list need to touch the tail pointer.
But of course, now that everything's unsafe pointers we need to rewrite the code to use those! And if we're going to be touching all the code, we might as well take the chance to make sure we aren't missing something.
But anyway, let's start copy-pasting code from the stack implementation:
// ...
pub struct IntoIter<T>(List<T>);
pub struct Iter<'a, T> {
next: Option<&'a Node<T>>,
}
pub struct IterMut<'a, T> {
next: Option<&'a mut Node<T>>,
}
IntoIter looks fine, but Iter and IterMut are breaking our simple rule of never using safe pointers in our types anymore. Let's be safe and change those to use raw pointers:
pub struct IntoIter<T>(List<T>);
pub struct Iter<'a, T> {
next: *mut Node<T>,
}
pub struct IterMut<'a, T> {
next: *mut Node<T>,
}
impl<T> List<T> {
pub fn into_iter(self) -> IntoIter<T> {
IntoIter(self)
}
pub fn iter(&self) -> Iter<'_, T> {
Iter { next: self.head }
}
pub fn iter_mut(&mut self) -> IterMut<'_, T> {
IterMut { next: self.head }
}
}
Looks good!
error[E0392]: parameter `'a` is never used
--> src\fifth.rs:17:17
|
17 | pub struct Iter<'a, T> {
| ^^ unused parameter
|
= help: consider removing `'a`, referring to it in a field,
or using a marker such as `PhantomData`
error[E0392]: parameter `'a` is never used
--> src\fifth.rs:21:20
|
21 | pub struct IterMut<'a, T> {
| ^^ unused parameter
|
= help: consider removing `'a`, referring to it in a field,
or using a marker such as `PhantomData`
Doesn't look good! What's this PhantomData they're on about?
Zero-sized type used to mark things that “act like” they own a
T.Adding a
PhantomData<T>field to your type tells the compiler that your type acts as though it stores a value of typeT, even though it doesn’t really. This information is used when computing certain safety properties.For a more in-depth explanation of how to use
PhantomData<T>, please see the Nomicon.
Hey don't get hasty there, we're reading the book that I wrote. Not that other book that some huge nerd probably wrote! I bet if they write a data structure in there it's something lame like an Array Stack and not a Linked List.
Unused lifetime parameters
Perhaps the most common use case for PhantomData is a struct that has an unused lifetime parameter, typically as part of some unsafe code.
Ah so we're naming a lifetime in our type but not actually using it. We could go down the PhantomData path, but I want to save that for the doubly-linked list in the next chapter that will really need it.
We're in an interesting situation where we actually don't need PhantomData. I think. I'm just going to claim that and trust that it's true, and if miri yells at us at the end I'll concede the point and we'll do the PhantomData thing.
What we're actually going to do is put the references back in these Iterator types and be happy we get to use references in some places still. I think that's sound because there's still a kind of proper nesting when you use an iterator: you create the iterator, use safe references for a while, and then discard the iterator.
Only once the iterator is gone can you access the list and call things like push and pop which need to mess with the tail pointer and Boxes. Now, during the iteration we are going to be dereferencing a bunch of raw pointers, so there is a kind of mixing there, but we should be able to think of those references as reborrows of the unsafe pointers.
I'm not even 100% convinced but I just wanna give it a try and see!
pub struct IntoIter<T>(List<T>);
pub struct Iter<'a, T> {
next: Option<&'a Node<T>>,
}
pub struct IterMut<'a, T> {
next: Option<&'a mut Node<T>>,
}
impl<T> List<T> {
pub fn into_iter(self) -> IntoIter<T> {
IntoIter(self)
}
pub fn iter(&self) -> Iter<'_, T> {
unsafe {
Iter { next: self.head.as_ref() }
}
}
pub fn iter_mut(&mut self) -> IterMut<'_, T> {
unsafe {
IterMut { next: self.head.as_mut() }
}
}
}
If we're going to be storing references, we need to upgrade our raw pointers to options-of-references. We could check if the pointer is null, but this is one of the incredibly narrow cases where I think it's ok to use the nasty ptr::as_ref and ptr::as_mut methods.
I usually recommend avoiding these methods like the plague because they do some surprising and nasty stuff and they're inherently reintroducing references when my whole "easy rule" is to avoid doing that!
Those methods come with a lot of warnings, but the most interesting is this:
You must enforce Rust’s aliasing rules, since the returned lifetime
'ais arbitrarily chosen and does not necessarily reflect the actual lifetime of the data. In particular, for the duration of this lifetime, the memory the pointer points to must not get accessed (read or written) through any other pointer.
Hey look it's the thing we talked about for 25 pages! I have already asserted we're definitely going to be fine to use references here, so aliasing solved! The other evil part is the signature:
pub unsafe fn as_mut<'a>(self) -> Option<&'a mut T>
Do you see how that lifetime isn't attached to the input at all, because self is by-value? Yeah that's what we call an "unbounded lifetime" and it's nasty stuff. It's willing to pretend to be as large as we ask it to be, even 'static! The way you deal with that is by putting it somewhere that is bounded, which usually just means "return this from a function as soon as possible so that the function signature limits it".
Boy I'm nervous about this but we're gonna keep pushing through! Let's steal some iterator impls from the stack:
impl<T> Iterator for IntoIter<T> {
type Item = T;
fn next(&mut self) -> Option<Self::Item> {
self.0.pop()
}
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&mut self) -> Option<Self::Item> {
unsafe {
self.next.map(|node| {
self.next = node.next.as_ref();
&node.elem
})
}
}
}
impl<'a, T> Iterator for IterMut<'a, T> {
type Item = &'a mut T;
fn next(&mut self) -> Option<Self::Item> {
unsafe {
self.next.take().map(|node| {
self.next = node.next.as_mut();
&mut node.elem
})
}
}
}
Moment of truth time...
cargo test
running 15 tests
test fifth::test::basics ... ok
test fifth::test::into_iter ... ok
test fifth::test::iter ... ok
test fifth::test::iter_mut ... ok
test first::test::basics ... ok
test fourth::test::basics ... ok
test fourth::test::into_iter ... ok
test fourth::test::peek ... ok
test second::test::basics ... ok
test second::test::into_iter ... ok
test second::test::iter ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test third::test::iter ... ok
test third::test::basics ... ok
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out;
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri test
running 15 tests
test fifth::test::basics ... ok
test fifth::test::into_iter ... ok
test fifth::test::iter ... ok
test fifth::test::iter_mut ... ok
test first::test::basics ... ok
test fourth::test::basics ... ok
test fourth::test::into_iter ... ok
test fourth::test::peek ... ok
test second::test::basics ... ok
test second::test::into_iter ... ok
test second::test::iter ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test third::test::basics ... ok
test third::test::iter ... ok
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
YES!!! Take that NARRATOR! Sometimes I don't make mistakes!
NARRATOR: but wasn't the whole point that the mistakes are there to teach the reader.
YEAH WELL SOMETIMES THE LESSON IS THAT I'M RIGHT AND EVERYONE SHOULD LISTEN TO ME WHEN I SAY THINGS ABOUT UNSAFE CODE BECAUSE I HAVE SPENT FAR TOO MUCH TIME THINKING ABOUT THE SOUNDNESS OF ITERATOR IMPLEMENTATIONS?! OK?! OK.
Anyway here's peek and peek_mut.
pub fn peek(&self) -> Option<&T> {
unsafe {
self.head.as_ref()
}
}
pub fn peek_mut(&mut self) -> Option<&mut T> {
unsafe {
self.head.as_mut()
}
}
I'm not even gonna test them because I never make mistakes anymore.
NARRATOR:
cargo build
error[E0308]: mismatched types
--> src\fifth.rs:66:13
|
25 | impl<T> List<T> {
| - this type parameter
...
64 | pub fn peek(&self) -> Option<&T> {
| ---------- expected `Option<&T>`
| because of return type
65 | unsafe {
66 | self.head.as_ref()
| ^^^^^^^^^^^^^^^^^^ expected type parameter `T`,
| found struct `fifth::Node`
|
= note: expected enum `Option<&T>`
found enum `Option<&fifth::Node<T>>`
FINE.
pub fn peek(&self) -> Option<&T> {
unsafe {
self.head.as_ref().map(|node| &node.elem)
}
}
pub fn peek_mut(&mut self) -> Option<&mut T> {
unsafe {
self.head.as_mut().map(|node| &mut node.elem)
}
}
I guess I am going to continue to make mistakes, so we're going to be extra careful and add a new test I'm going to call "miri food": something that just messes around and mixes up our APIs a bunch to help miri catch our mistakes.
#[test]
fn miri_food() {
let mut list = List::new();
list.push(1);
list.push(2);
list.push(3);
assert!(list.pop() == Some(1));
list.push(4);
assert!(list.pop() == Some(2));
list.push(5);
assert!(list.peek() == Some(&3));
list.push(6);
list.peek_mut().map(|x| *x *= 10);
assert!(list.peek() == Some(&30));
assert!(list.pop() == Some(30));
for elem in list.iter_mut() {
*elem *= 100;
}
let mut iter = list.iter();
assert_eq!(iter.next(), Some(&400));
assert_eq!(iter.next(), Some(&500));
assert_eq!(iter.next(), Some(&600));
assert_eq!(iter.next(), None);
assert_eq!(iter.next(), None);
assert!(list.pop() == Some(400));
list.peek_mut().map(|x| *x *= 10);
assert!(list.peek() == Some(&5000));
list.push(7);
// Drop it on the ground and let the dtor exercise itself
}
cargo test
running 16 tests
test fifth::test::basics ... ok
test fifth::test::into_iter ... ok
test fifth::test::iter ... ok
test fifth::test::iter_mut ... ok
test fifth::test::miri_food ... ok
test first::test::basics ... ok
test fourth::test::basics ... ok
test fourth::test::into_iter ... ok
test fourth::test::peek ... ok
test second::test::into_iter ... ok
test second::test::basics ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test third::test::iter ... ok
test second::test::iter ... ok
test third::test::basics ... ok
test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
MIRIFLAGS="-Zmiri-tag-raw-pointers" cargo +nightly-2022-01-21 miri test
running 16 tests
test fifth::test::basics ... ok
test fifth::test::into_iter ... ok
test fifth::test::iter ... ok
test fifth::test::iter_mut ... ok
test fifth::test::miri_food ... ok
test first::test::basics ... ok
test fourth::test::basics ... ok
test fourth::test::into_iter ... ok
test fourth::test::peek ... ok
test second::test::into_iter ... ok
test second::test::basics ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test third::test::iter ... ok
test second::test::iter ... ok
test third::test::basics ... ok
test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Perfect.
Final Code
Alright, so with a teeny-tiny dash of unsafety we managed to get a linear time improvement over the naive safe queue, and we managed to reuse almost all of the logic from the safe stack!
You know, except for that part where miri completely dunked on us and we had to write a short master's thesis on rust's memory model. You know, as you do.
But on the bright side we didn't have to write any jank Rc or RefCell stuff.
#![allow(unused)] fn main() { use std::ptr; pub struct List<T> { head: Link<T>, tail: *mut Node<T>, } type Link<T> = *mut Node<T>; struct Node<T> { elem: T, next: Link<T>, } pub struct IntoIter<T>(List<T>); pub struct Iter<'a, T> { next: Option<&'a Node<T>>, } pub struct IterMut<'a, T> { next: Option<&'a mut Node<T>>, } impl<T> List<T> { pub fn new() -> Self { List { head: ptr::null_mut(), tail: ptr::null_mut() } } pub fn push(&mut self, elem: T) { unsafe { let new_tail = Box::into_raw(Box::new(Node { elem: elem, next: ptr::null_mut(), })); if !self.tail.is_null() { (*self.tail).next = new_tail; } else { self.head = new_tail; } self.tail = new_tail; } } pub fn pop(&mut self) -> Option<T> { unsafe { if self.head.is_null() { None } else { let head = Box::from_raw(self.head); self.head = head.next; if self.head.is_null() { self.tail = ptr::null_mut(); } Some(head.elem) } } } pub fn peek(&self) -> Option<&T> { unsafe { self.head.as_ref().map(|node| &node.elem) } } pub fn peek_mut(&mut self) -> Option<&mut T> { unsafe { self.head.as_mut().map(|node| &mut node.elem) } } pub fn into_iter(self) -> IntoIter<T> { IntoIter(self) } pub fn iter(&self) -> Iter<'_, T> { unsafe { Iter { next: self.head.as_ref() } } } pub fn iter_mut(&mut self) -> IterMut<'_, T> { unsafe { IterMut { next: self.head.as_mut() } } } } impl<T> Drop for List<T> { fn drop(&mut self) { while let Some(_) = self.pop() { } } } impl<T> Iterator for IntoIter<T> { type Item = T; fn next(&mut self) -> Option<Self::Item> { self.0.pop() } } impl<'a, T> Iterator for Iter<'a, T> { type Item = &'a T; fn next(&mut self) -> Option<Self::Item> { unsafe { self.next.map(|node| { self.next = node.next.as_ref(); &node.elem }) } } } impl<'a, T> Iterator for IterMut<'a, T> { type Item = &'a mut T; fn next(&mut self) -> Option<Self::Item> { unsafe { self.next.take().map(|node| { self.next = node.next.as_mut(); &mut node.elem }) } } } #[cfg(test)] mod test { use super::List; #[test] fn basics() { let mut list = List::new(); // Check empty list behaves right assert_eq!(list.pop(), None); // Populate list list.push(1); list.push(2); list.push(3); // Check normal removal assert_eq!(list.pop(), Some(1)); assert_eq!(list.pop(), Some(2)); // Push some more just to make sure nothing's corrupted list.push(4); list.push(5); // Check normal removal assert_eq!(list.pop(), Some(3)); assert_eq!(list.pop(), Some(4)); // Check exhaustion assert_eq!(list.pop(), Some(5)); assert_eq!(list.pop(), None); // Check the exhaustion case fixed the pointer right list.push(6); list.push(7); // Check normal removal assert_eq!(list.pop(), Some(6)); assert_eq!(list.pop(), Some(7)); assert_eq!(list.pop(), None); } #[test] fn into_iter() { let mut list = List::new(); list.push(1); list.push(2); list.push(3); let mut iter = list.into_iter(); assert_eq!(iter.next(), Some(1)); assert_eq!(iter.next(), Some(2)); assert_eq!(iter.next(), Some(3)); assert_eq!(iter.next(), None); } #[test] fn iter() { let mut list = List::new(); list.push(1); list.push(2); list.push(3); let mut iter = list.iter(); assert_eq!(iter.next(), Some(&1)); assert_eq!(iter.next(), Some(&2)); assert_eq!(iter.next(), Some(&3)); assert_eq!(iter.next(), None); } #[test] fn iter_mut() { let mut list = List::new(); list.push(1); list.push(2); list.push(3); let mut iter = list.iter_mut(); assert_eq!(iter.next(), Some(&mut 1)); assert_eq!(iter.next(), Some(&mut 2)); assert_eq!(iter.next(), Some(&mut 3)); assert_eq!(iter.next(), None); } #[test] fn miri_food() { let mut list = List::new(); list.push(1); list.push(2); list.push(3); assert!(list.pop() == Some(1)); list.push(4); assert!(list.pop() == Some(2)); list.push(5); assert!(list.peek() == Some(&3)); list.push(6); list.peek_mut().map(|x| *x *= 10); assert!(list.peek() == Some(&30)); assert!(list.pop() == Some(30)); for elem in list.iter_mut() { *elem *= 100; } let mut iter = list.iter(); assert_eq!(iter.next(), Some(&400)); assert_eq!(iter.next(), Some(&500)); assert_eq!(iter.next(), Some(&600)); assert_eq!(iter.next(), None); assert_eq!(iter.next(), None); assert!(list.pop() == Some(400)); list.peek_mut().map(|x| *x *= 10); assert!(list.peek() == Some(&5000)); list.push(7); // Drop it on the ground and let the dtor exercise itself } } }
A Production-Quality Unsafe Doubly-Linked Deque
We finally made it. My greatests nemesis: std::collections::LinkedList, the Doubly-Linked Deque.
The one that I tried and failed to destroy.
Our story begins as 2014 was coming to a close and we were rapidly approaching the release of Rust 1.0, Rust's first stable release. I had found myself in the role of caring for std::collections, or as we affectionately called it in those times, libcollections.
libcollections had spent years as a dumping ground for everyone's Cute Ideas and Vaguely Useful Things. This was all well and good when Rust was a fledgling experimental language, but if my children were going to escape the nest and be stabilized, they would have to prove their worth.
Until then I had encouraged and nurtured them all, but it was now time for them to face judgement for their failings.
I sunk my claws into the bedrock and carved tombstones for my most foolish children. A grisly monument that I placed in the town square for all to see:
Kill TreeMap, TreeSet, TrieMap, TrieSet, LruCache and EnumSet
Their fates were sealed, for my word was absolute. The other collections were horrified by my brutality, but they were not yet safe from their mother's wrath. I soon returned with two more tombstones:
The Bit twins were more cunning than their fallen comrades, but they lacked the strength to escape me. Most thought my work done, but I soon took one more:
VecMap had tried to survive through stealth — it was so small and inoffensive! But that wasn't enough for the libcollections I saw in my vision of the future.
I surveyed the land and saw what remained:
- Vec and VecDeque - hearty and simple, the heart of computing.
- HashMap and HashSet - powerful and wise, the brain of computing.
- BTreeMap and BTreeSet - awkward but necessary, the liver of computing.
- BinaryHeap - crafty and dextrous, the ankle of computing.
I nodded in contentment. Simple and effective. My work was don—
No, DList, it can't be! I thought you died in that tragic garbage collection incident! The one which was definitely an accident and not intentional at all!
They had faked their death and taken on a new name, but it was still them: LinkedList, the shadowy and untrustworthy schemer of computing.
I spread word of their misdeeds to all that would hear me, but hearts were unmoved. LinkedList was a silver-tongued devil who had convinced everyone around me that it was some sort of fundamental and natural datastructure of computing. It had even convinced C++ that it was the list!
"How could you have a standard library without a LinkedList?"
Easily! Trivially!
"It's non-trivial unsafe code, so it makes sense to have it in the standard library!"
So are GPU drivers and video codecs, libcollections is minimalist!
But alas, LinkedList had gathered too many allies and grown too strong while I was distracted with its kin.
I fled to my laboratory and tried to devise some sort of evil clone or enhanced cyborg replicant that could rival and destroy it, but my grant funding ran out because my research was "too murderously evil" or somesuch nonsense.
LinkedList had won. I was defeated and forced into exile.
But you're here now. You've come this far. Surely now you can understand the depths of LinkedList's debauchery! Come, I will you show you everything you need to know to help me destroy it once and for all — everything you need to know to implement an unsafe production-quality Doubly-Linked Deque.
How production-quality? Well we're going to completely rewrite my ancient Rust 1.0 linked-list crate, the one that is objectively better than the one in std. The one with Cursors on stable Rust, from 2015! Something the 2022 stdlib still doesn't have!
Layout
Let us begin by first studying the structure of our enemy. A Doubly-Linked List is conceptually simple, but that's how it decieves and manipulates you. It's the same kind of linked list we've looked at over and over, but the links go both ways. Double the links, double the evil.
So rather than this (gonna drop the Some/None stuff to keep it cleaner):
... -> (A, ptr) -> (B, ptr) -> ...
We have this:
... <-> (ptr, A, ptr) <-> (ptr, B, ptr) <-> ...
This lets you traverse the list from either direction, or seek back and forth with a cursor.
In exchange for this flexibility, every node has to store twice as many pointers, and every operation has to fix up way more pointers. It's a significant enough complication that it's a lot easier to make a mistake, so we're going to be doing a lot of testing.
You might have also noticed that I intentionally haven't drawn the ends of the list. This is because this is one of the places where there are genuinely defensible options for our implementation. We definitely need our implementation to have two pointers: one to the start of the list, and one to the end of the list.
There are two notable ways to do this in my mind: "traditional" and "dummy node".
The traditional approach is the simple extension of how we did a Stack — just store the head and tail pointers on the stack:
[ptr, ptr] <-> (ptr, A, ptr) <-> (ptr, B, ptr)
^ ^
+----------------------------------------+
This is fine, but it has one downside: corner cases. There are now two edges to our list, which means twice as many corner cases. It's easy to forget one and have a serious bug.
The dummy node approach attempts to smooth out these corner cases by adding an extra node to our list which contains no data but links the two ends together into a ring:
[ptr] -> (ptr, ?DUMMY?, ptr) <-> (ptr, A, ptr) <-> (ptr, B, ptr)
^ ^
+-------------------------------------------------+
By doing this, every node always has actual pointers to a previous and next node in the list. Even when you remove the last element from the list, you just end up stitching the dummy node to point at itself:
[ptr] -> (ptr, ?DUMMY?, ptr)
^ ^
+-------------+
There is a part of me that finds this very satisfying and elegant. Unfortunately, it has a couple practical problems:
Problem 1: An extra indirection and allocation, especially for the empty list, which must include the dummy node. Potential solutions include:
-
Don't allocate the dummy node until something is inserted: simple and effective, but it adds back some of the corner cases we were trying to avoid by using dummy pointers!
-
Use a static copy-on-write empty singleton dummy node, with some really clever scheme that lets the Copy-On-Write checks piggy-back on normal checks: look I'm really tempted, I really do love that shit, but we can't go down that dark path in this book. Read ThinVec's sourcecode if you want to see that kind of perverted stuff.
-
Store the dummy node on the stack - not practical in a language without C++-style move-constructors. I'm sure there's something weird thing we could do here with pinning but we're not gonna.
Problem 2: What value is stored in the dummy node? Sure if it's an integer it's fine, but what if we're storing a list full of Box? It may be impossible for us to initialized this value! Potential solutions include:
-
Make every node store
Option<T>: simple and effective, but also bloated and annoying. -
Make every node store
MaybeUninit<T>. Horrifying and annoying. -
Really careful and clever inheritance-style type punning so the dummy node doesn't include the data field. This is also tempting but it's extremely dangerous and annoying. Read BTreeMap's source if you want to see that kind of perverted stuff.
The problems really outweigh the convenience for a language like Rust, so we're going to stick to the traditional layout. We'll be using the same basic design as we did for the unsafe queue in the previous chapter:
#![allow(unused)] fn main() { pub struct LinkedList<T> { front: Link<T>, back: Link<T>, len: usize, } type Link<T> = *mut Node<T>; struct Node<T> { front: Link<T>, back: Link<T>, elem: T, } }
(Now that we have reached the doubly-linked-deque, we have finally earned the right to call ourselves LinkedList, for this is the True Linked List.)
This isn't quite a true production-quality layout yet. It's fine but there's magic tricks we can do to tell Rust what we're doing a bit better. To do that we're going to need to go... deeper.
Variance and PhantomData
It's going to be annoying to punt on this now and fix it later, so we're going to do the Hardcore Layout stuff now.
There are five terrible horsemen of making unsafe Rust collections:
Mercifully, the last 2 aren't going to be a problem for us.
The third we could make into our problem but it's more trouble than it's worth -- if you've opted into a LinkedList you've already given up the battle on memory-effeciency 100-fold already.
The second is something that I used to insist was really important and that std messes around with, but the defaults are safe, the ways to mess with it are unstable, and you need to try so very hard to ever notice the limitations of the defaults, so, don't worry about it.
That just leaves us with Variance. To be honest, you can probably punt on this one too, but I still have my pride as a Collections Person, so we're going to Do The Variance Thing.
So, surprise: Rust has subtyping. In particular, &'big T is a subtype of &'small T. Why? Well because if some code needs a reference that lives for some particular region of the program, it's usually perfectly fine to give it a reference that lives for longer. Like, intuitively that's just true, right?
Why is this important? Well imagine some code that takes two values with the same type:
fn take_two<T>(_val1: T, _val2: T) { }
This is some deeply boring code, and so we should expect it to work with T=&u32 fine, right?
#![allow(unused)] fn main() { fn two_refs<'big: 'small, 'small>( big: &'big u32, small: &'small u32, ) { take_two(big, small); } fn take_two<T>(_val1: T, _val2: T) { } }
Yep, that compiles fine!
Now let's have some fun and wrap it in, oh, I don't know, std::cell::Cell:
#![allow(unused)] fn main() { use std::cell::Cell; fn two_refs<'big: 'small, 'small>( // NOTE: these two lines changed big: Cell<&'big u32>, small: Cell<&'small u32>, ) { take_two(big, small); } fn take_two<T>(_val1: T, _val2: T) { } }
error[E0623]: lifetime mismatch
--> src/main.rs:7:19
|
4 | big: Cell<&'big u32>,
| ---------
5 | small: Cell<&'small u32>,
| ----------- these two types are declared with different lifetimes...
6 | ) {
7 | take_two(big, small);
| ^^^^^ ...but data from `small` flows into `big` here
Huh??? We didn't touch the lifetimes, why's the compiler angry now!?
Ah well, the lifetime "subtyping" stuff must be really simple, so it falls over if you wrap the references in anything, see look it breaks with Vec too:
#![allow(unused)] fn main() { fn two_refs<'big: 'small, 'small>( big: Vec<&'big u32>, small: Vec<&'small u32>, ) { take_two(big, small); } fn take_two<T>(_val1: T, _val2: T) { } }
Finished dev [unoptimized + debuginfo] target(s) in 1.07s
Running `target/debug/playground`
See it doesn't compile eith-- wait what??? Vec is magic??????
Well, yes. But also, no. The magic was inside us all along, and that magic is ✨Variance✨.
Read the nomicon's chapter on subtyping if you want all the gorey details, but basically subtyping isn't always safe. In particular it's not safe when mutable references are involved because you can use things like mem::swap and suddenly oops dangling pointers!
Things that are "like mutable references" are invariant which means they block subtyping from happening on their generic parameters. So for safety, &mut T is invariant over T, and Cell<T> is invariant over T because &Cell<T> is basically just &mut T (because of interior mutability).
Almost everything that isn't invariant is covariant, and that just means that subtyping "passes through" it and continues to work normally (there are also contravariant types that make subtyping go backwards but they are really rare and no one likes them so I won't mention them again).
Collections generally contain a mutable pointer to their data, so you might expect them to be invariant too, but in fact, they don't need to be! Because of Rust's ownership system, Vec<T> is semantically equivalent to T, and that means it's safe for it to be covariant!
Unfortunately, this definition is invariant:
#![allow(unused)] fn main() { pub struct LinkedList<T> { front: Link<T>, back: Link<T>, len: usize, } type Link<T> = *mut Node<T>; struct Node<T> { front: Link<T>, back: Link<T>, elem: T, } }
But how is Rust actually deciding the variance of things? Well in the good-old-days before 1.0 we messed around with just letting people specify the variance they wanted and... it was an absolute train-wreck! Subtyping and variance is really hard to wrap your head around, and core developers genuinely disagreed on basic terminology! So we moved to a "variance by example" approach: the compiler just looks at your fields and copies their variances. If there's any kind of disagreement, then invariance always wins, because that's safe.
So what's in our type definitions that Rust is getting mad about? *mut!
Raw pointers in Rust really just try to let you do whatever, but they have exactly one safety feature: because most people have no idea that variance and subtyping are a thing in Rust, and being incorrectly covariant would be horribly dangerous, *mut T is invariant, because there's a good chance it's being used "as" &mut T.
This is extremely annoying for Exactly Me as a person who has spent a lot of time writing collections in Rust. This is why when I made std::ptr::NonNull, I added this little piece of magic:
Unlike
*mut T,NonNull<T>was chosen to be covariant overT. This makes it possible to useNonNull<T>when building covariant types, but introduces the risk of unsoundness if used in a type that shouldn’t actually be covariant.
But hey, it's interface is built around *mut T, what's the deal! Is it just magic?! Let's look:
#![allow(unused)] fn main() { pub struct NonNull<T> { pointer: *const T, } impl<T> NonNull<T> { pub unsafe fn new_unchecked(ptr: *mut T) -> Self { // SAFETY: the caller must guarantee that `ptr` is non-null. unsafe { NonNull { pointer: ptr as *const T } } } } }
NOPE. NO MAGIC HERE! NonNull just abuses the fact that *const T is covariant and stores that instead, casting back and forth between *mut T at the API boundary to make it "look like" it's storing a *mut T. That's the whole trick! That's how collections in Rust are covariant! And it's miserable! So I made the Good Pointer Type do it for you! You're welcome! Enjoy your subtyping footgun!
The solution to all your problems is to use NonNull, and then if you want to have nullable pointers again, use Option<NonNull<T>>. Are we really going to bother doing that..?
Yep! It sucks, but we're making production grade linked lists so we're going to eat all our vegetables and do things the hard way (we could just use bare *const T and cast everywhere, but I genuinely want to see how painful this is... for Ergonomics Science).
So here's our final type definitions:
#![allow(unused)] fn main() { use std::ptr::NonNull; // !!!This changed!!! pub struct LinkedList<T> { front: Link<T>, back: Link<T>, len: usize, } type Link<T> = Option<NonNull<Node<T>>>; struct Node<T> { front: Link<T>, back: Link<T>, elem: T, } }
...wait nope, one last thing. Any time you do raw pointer stuff, you should add a Ghost to protect your pointers:
use std::marker::PhantomData;
pub struct LinkedList<T> {
front: Link<T>,
back: Link<T>,
len: usize,
/// We semantically store values of T by-value.
_boo: PhantomData<T>,
}
In this case I don't think we actually need PhantomData, but any time you do use NonNull (or just raw pointers in general), you should always add it to be safe and make it clear to the compiler and others what you think you're doing.
PhantomData is a way for us to give the compiler an extra "example" field that conceptually exists in your type but for various reasons (indirection, type erasure, ...) doesn't. In this case we're using NonNull because we're claiming our type behaves "as if" it stored a value T, so we add a PhantomData to make that explicit.
The stdlib actually has other reasons to do this because it has access to the accursed Drop Check overrides, but that feature has been reworked so many times that I don't actually know if the PhantomData thing is a thing for it anymore. I'm still going to cargo-cult it for all eternity, because Drop Check Magic is burned into my brain!
(Node literally stores a T, so it doesn't have to do this, yay!)
...ok for real we're done with layout now! On to actual basic functionality!
Basics
Alright, this is the part of the book that sucks, and why it took me 7 years to write this chapter! Time to just burn through a whole lot of really boring stuff we've done 5 times already, but extra verbose and long because we have to do everything twice and with Option<NonNull<Node<T>>>!
impl<T> LinkedList<T> {
pub fn new() -> Self {
Self {
front: None,
back: None,
len: 0,
_boo: PhantomData,
}
}
}
PhantomData is a weird type with no fields so you just make one by, saying its type name. shrug
pub fn push_front(&mut self, elem: T) {
// SAFETY: it's a linked-list, what do you want?
unsafe {
let new = NonNull::new_unchecked(Box::into_raw(Box::new(Node {
front: None,
back: None,
elem,
})));
if let Some(old) = self.front {
// Put the new front before the old one
(*old).front = Some(new);
(*new).back = Some(old);
} else {
// If there's no front, then we're the empty list and need
// to set the back too. Also here's some integrity checks
// for testing, in case we mess up.
debug_assert!(self.back.is_none());
debug_assert!(self.front.is_none());
debug_assert!(self.len == 0);
self.back = Some(new);
}
self.front = Some(new);
self.len += 1;
}
}
error[E0614]: type `NonNull<Node<T>>` cannot be dereferenced
--> src\lib.rs:39:17
|
39 | (*old).front = Some(new);
| ^^^^^^
Ah yes, I truly hate my pointer-y children. We need to explicitly get the raw pointer out of NonNull with as_ptr, because DerefMut is defined in terms of &mut and we don't want to randomly introduce safe references into our unsafe code!
(*old.as_ptr()).front = Some(new);
(*new.as_ptr()).back = Some(old);
Compiling linked-list v0.0.3
warning: field is never read: `elem`
--> src\lib.rs:16:5
|
16 | elem: T,
| ^^^^^^^
|
= note: `#[warn(dead_code)]` on by default
warning: `linked-list` (lib) generated 1 warning (1 duplicate)
warning: `linked-list` (lib test) generated 1 warning
Finished test [unoptimized + debuginfo] target(s) in 0.33s
Nice, now for pop (and len):
pub fn pop_front(&mut self) -> Option<T> {
unsafe {
// Only have to do stuff if there is a front node to pop.
// Note that we don't need to mess around with `take` anymore
// because everything is Copy and there are no dtors that will
// run if we mess up... right? :) Riiiight? :)))
self.front.map(|node| {
// Bring the Box back to life so we can move out its value and
// Drop it (Box continues to magically understand this for us).
let boxed_node = Box::from_raw(node.as_ptr());
let result = boxed_node.elem;
// Make the next node into the new front.
self.front = boxed_node.back;
if let Some(new) = self.front {
// Cleanup its reference to the removed node
(*new.as_ptr()).front = None;
} else {
// If the front is now null, then this list is now empty!
debug_assert!(self.len == 1);
self.back = None;
}
self.len -= 1;
result
// Box gets implicitly freed here, knows there is no T.
})
}
}
pub fn len(&self) -> usize {
self.len
}
Compiling linked-list v0.0.3
Finished dev [unoptimized + debuginfo] target(s) in 0.37s
Seems legit to me, time to write a test!
#[cfg(test)]
mod test {
use super::LinkedList;
#[test]
fn test_basic_front() {
let mut list = LinkedList::new();
// Try to break an empty list
assert_eq!(list.len(), 0);
assert_eq!(list.pop_front(), None);
assert_eq!(list.len(), 0);
// Try to break a one item list
list.push_front(10);
assert_eq!(list.len(), 1);
assert_eq!(list.pop_front(), Some(10));
assert_eq!(list.len(), 0);
assert_eq!(list.pop_front(), None);
assert_eq!(list.len(), 0);
// Mess around
list.push_front(10);
assert_eq!(list.len(), 1);
list.push_front(20);
assert_eq!(list.len(), 2);
list.push_front(30);
assert_eq!(list.len(), 3);
assert_eq!(list.pop_front(), Some(30));
assert_eq!(list.len(), 2);
list.push_front(40);
assert_eq!(list.len(), 3);
assert_eq!(list.pop_front(), Some(40));
assert_eq!(list.len(), 2);
assert_eq!(list.pop_front(), Some(20));
assert_eq!(list.len(), 1);
assert_eq!(list.pop_front(), Some(10));
assert_eq!(list.len(), 0);
assert_eq!(list.pop_front(), None);
assert_eq!(list.len(), 0);
assert_eq!(list.pop_front(), None);
assert_eq!(list.len(), 0);
}
}
Compiling linked-list v0.0.3
Finished test [unoptimized + debuginfo] target(s) in 0.40s
Running unittests src\lib.rs
running 1 test
test test::test_basic_front ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Hooray, we're perfect!
...Right?
Drop and Panic Safety
So hey, did you notice this comment:
#![allow(unused)] fn main() { // Note that we don't need to mess around with `take` anymore // because everything is Copy and there are no dtors that will // run if we mess up... right? :) Riiiight? :))) }
Is it right?
Sorry did you forget the book you're reading? Of course it's wrong! (Sort Of.)
Let's look at the inner body of pop_front again:
// Bring the Box back to life so we can move out its value and
// Drop it (Box continues to magically understand this for us).
let boxed_node = Box::from_raw(node.as_ptr());
let result = boxed_node.elem;
// Make the next node into the new front.
self.front = boxed_node.back;
if let Some(new) = self.front {
// Cleanup its reference to the removed node
(*new.as_ptr()).front = None;
} else {
// If the front is now null, then this list is now empty!
debug_assert!(self.len == 1);
self.back = None;
}
self.len -= 1;
result
// Box gets implicitly freed here, knows there is no T.
Do you see the bug? Horrifyingly, it's actually this line:
debug_assert!(self.len == 1);
Really? Our friggin' integrity check for tests is a bug?? Yes!!! Well, if we implement our collection right it shouldn't be, but it can turn something benign like "oh we are doing a bad job of keeping len up to date" into An Exploitable Memory Safety Bug! Why? Because it can panic! Most of the time you don't have to think or worry about panics, but once you start writing really unsafe code and playing fast and loose with "invariants", you need to become hyper-vigilant about panics!
We've gotta talk about exception safety (AKA panic safety, AKA unwind safety, ...).
So here's the deal: by default, panics are unwinding. Unwinding is just a fancy way to say "make every single function immediately return". You might think "well, if everyone returns then the program is about to die, so why care about it?", but you'd be wrong!
We have to care for two reasons: destructors run when a function returns, and the unwind can be caught. In both cases, code can keep running after a panic, so we need to be very careful and make sure our unsafe collections are always in some kind of coherent state whenever a panic could occur, because each panic is an implicit early return!
Let's think about what state our collection is in when we get to that line:
We have our boxed_node on the stack, and we've extracted the element from it. If we were to return at this point, the Box would be dropped, and the node would be freed. Do you see it now..? self.back is still pointing at that freed node! Once we implement the rest of our collection and start using self.back for things, this could result in a use-after-free! Yikes!
Interestingly, this line has similar problems, but it's much safer:
self.len -= 1;
By default in debug builds Rust checks for underflows and overflows and will panic when they happen. Yes, every arithmetic operation is a panic-safety hazard! This one is better because it happens after we've repaired all of our invariants, so it won't cause memory-safety issues... as long as we don't trust len to be right, but then, if we underflow it's definitely wrong, so we were dead either way! The debug assert is in some sense worse because it can escalate a minor issue into a critical one!
I've brought up the term "invariants" a few times, and that's because it's a really useful concept for panic-safety! Basically, to an outside observer of our collection, there are certain property we're always upholding. For a LinkedList, one of those is that any node that is reachable in our list is still allocated and initialized.
Inside the implementation we have a bit more flexibility to break invariants temporarily as long as we make sure to repair them before anyone notices. This is actually one of the "killer apps" of Rust's ownership and borrowing system for collections: if an operation requires an &mut Self, then we are guaranteed that we have exclusive access to our collection and that it's fine for us to temporarily break invariants, safe in the knowledge that no one can sneakily mess with it.
Perhaps the greatest expression of this is Vec::drain, which actually lets you completely smash a core invariant of Vec and start moving values out from the front or even middle of a Vec. The reason this is sound is because the Drain iterator that we return holds an &mut to the Vec, and so all access is gated behind it! No one can observe the Vec until the Drain iterator goes away, and then it's destructor can "repair" the Vec before anyone can notice, it's perfe--
It's not perfect. Unfortunately, you can't rely on destructors in code you don't control to run, and so even with Drain we need to do a little extra work to make our type always preserved invariants, but in a kind of goofy way: we just set the Vec's len to 0 at the start, so if anyone leaks the Drain, then they will have a safe Vec... but they will have also lost a bunch of data. You leak me? I leak you! An eye for an eye! True justice!
For a situation where you can actually use destructors for panic-safety, check out the BinaryHeap::sift_up case study.
Anyway, we won't be needing all of this fancy stuff for our LinkedLists, we just need to be a bit more vigilant about where we break our invariants, what we trust/require to be correct, and to avoid introducing unnecessary unwinds in the middle of hairy tasks.
In this case, we have two options to make our code a bit more robust:
-
Use operations like Option::take a lot more aggressively, because they are more "transactional" and have a tendency to preserve invariants.
-
Kill the debug_asserts and trust ourselves to write better tests with dedicated "integrity check" functions that won't run in user code ever.
In principle I like the first option, but it doesn't actually work great for a doubly-linked list, because everything is doubly-redundantly encoded. Option::take wouldn't fix the problem here, but moving the debug_assert down a line would. But really, why make things harder for ourselves? Let's just remove those debug_asserts, and make sure anything can panic is at the start or end of our methods, where our invariants should be known to hold.
(In this way it's perhaps more accurate to think of them as preconditions and postconditions but you really should endeavour to treat them as invariants as much as possible!)
Here's our full implementation now:
#![allow(unused)] fn main() { use std::ptr::NonNull; use std::marker::PhantomData; pub struct LinkedList<T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<T>, } type Link<T> = Option<NonNull<Node<T>>>; struct Node<T> { front: Link<T>, back: Link<T>, elem: T, } impl<T> LinkedList<T> { pub fn new() -> Self { Self { front: None, back: None, len: 0, _boo: PhantomData, } } pub fn push_front(&mut self, elem: T) { // SAFETY: it's a linked-list, what do you want? unsafe { let new = NonNull::new_unchecked(Box::into_raw(Box::new(Node { front: None, back: None, elem, }))); if let Some(old) = self.front { // Put the new front before the old one (*old.as_ptr()).front = Some(new); (*new.as_ptr()).back = Some(old); } else { // If there's no front, then we're the empty list and need // to set the back too. self.back = Some(new); } // These things always happen! self.front = Some(new); self.len += 1; } } pub fn pop_front(&mut self) -> Option<T> { unsafe { // Only have to do stuff if there is a front node to pop. self.front.map(|node| { // Bring the Box back to life so we can move out its value and // Drop it (Box continues to magically understand this for us). let boxed_node = Box::from_raw(node.as_ptr()); let result = boxed_node.elem; // Make the next node into the new front. self.front = boxed_node.back; if let Some(new) = self.front { // Cleanup its reference to the removed node (*new.as_ptr()).front = None; } else { // If the front is now null, then this list is now empty! self.back = None; } self.len -= 1; result // Box gets implicitly freed here, knows there is no T. }) } } pub fn len(&self) -> usize { self.len } } }
What can panic here? Well, knowing that honestly requires you to be a bit of a Rust expert, but thankfully, I am!
The only places I can see in this code that possibly can panic (barring some absolute fuckery where someone recompiles the stdlib with debug_asserts enabled, but this is not something you should ever do) are Box::new (for out-of-memory conditions) and the len arithmetic. All of that stuff is at the very end or very start of our methods, so yep, we're being nice and safe!
...were you surprised by Box::new being able to panic? Panics will get you like that! Try to preserve those invariants so you don't need to worry about it!
Boring Combinatorics
Ok, back to our regularly scheduled linked lists!
First let's knock out Drop which is trivial with pop:
impl<T> Drop for LinkedList<T> {
fn drop(&mut self) {
// Pop until we have to stop
while let Some(_) = self.pop_front() { }
}
}
We've got to fill in a bunch of really boring combinatoric implementations like front, front_mut, back, back_mut, iter, iter_mut, into_iter, ...
You could do them with macros or whatever but honestly, that's a worse fate than copy-pasting. We're just going to do a lot of copy-pasting. I have very carefully crafted the previous push/pop implementations so that we should be able to literally just swap front and back and the code does/says the right thing! Hooray for painful experience! (It's so tempting to talk about "prev and next" for nodes, but I find it's really worth it to just consistently talk about "front" and "back" as much as possible to avoid mistakes.)
Alright, first up, front:
pub fn front(&self) -> Option<&T> {
unsafe {
self.front.map(|node| &(*node.as_ptr()).elem)
}
}
Hey actually, this book is really old and some nice new things have been added like the ? operator which does an early return on Option::None, does that make our code nicer?
pub fn front(&self) -> Option<&T> {
unsafe {
Some(&(*self.front?.as_ptr()).elem)
}
}
Maybe? It's kind of a wash for something this simple, and the previous section was all about how early returns are kinda spooky for us, so maybe we should prefer being a bit more explicit here (I'm sticking to the map implementation). On to front_mut:
pub fn front_mut(&mut self) -> Option<&mut T> {
unsafe {
self.front.map(|node| &mut (*node.as_ptr()).elem)
}
}
I'll just dump all the back versions at the end.
Next up, iterators. Unlike all of our previous lists we've finally unlocked the ability to do DoubleEndedIterator, and if we're going for production quality we're gonna do ExactSizeIterator too.
So in addition to next and size_hint, we're going to support next_back and len.
The vigilant among you might notice that IterMut seems a lot more sketchy with double-ended iteration, but it's actually still sound!
... god this is gonna be a lot of boilerplate. Maybe I should really write a macro... no, no, that's still a worse fate.
pub struct Iter<'a, T> {
front: Link<T>,
back: Link<T>,
len: usize,
_boo: PhantomData<&'a T>,
}
impl<T> LinkedList<T> {
pub fn iter(&self) -> Iter<T> {
Iter {
front: self.front,
back: self.back,
len: self.len,
_boo: PhantomData,
}
}
}
impl<'a, T> IntoIterator for &'a LinkedList<T> {
type IntoIter = Iter<'a, T>;
type Item = &'a T;
fn into_iter(self) -> Self::IntoIter {
self.iter()
}
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&mut self) -> Option<Self::Item> {
// While self.front == self.back is a tempting condition to check here,
// it won't do the right for yielding the last element! That sort of
// thing only works for arrays because of "one-past-the-end" pointers.
if self.len > 0 {
// We could unwrap front, but this is safer and easier
self.front.map(|node| unsafe {
self.len -= 1;
self.front = (*node.as_ptr()).back;
&(*node.as_ptr()).elem
})
} else {
None
}
}
fn size_hint(&self) -> (usize, Option<usize>) {
(self.len, Some(self.len))
}
}
impl<'a, T> DoubleEndedIterator for Iter<'a, T> {
fn next_back(&mut self) -> Option<Self::Item> {
if self.len > 0 {
self.back.map(|node| unsafe {
self.len -= 1;
self.back = (*node.as_ptr()).front;
&(*node.as_ptr()).elem
})
} else {
None
}
}
}
impl<'a, T> ExactSizeIterator for Iter<'a, T> {
fn len(&self) -> usize {
self.len
}
}
...that's just .iter()...
we'll paste IterMut at the end, it's literally the exact same code with mut in a lot of places, let's just knock out into_iter first. We can mercifully still lean on our tried-and-true solution of just making it wrap our collection and using pop for next:
pub struct IntoIter<T> {
list: LinkedList<T>,
}
impl<T> LinkedList<T> {
pub fn into_iter(self) -> IntoIter<T> {
IntoIter {
list: self
}
}
}
impl<T> IntoIterator for LinkedList<T> {
type IntoIter = IntoIter<T>;
type Item = T;
fn into_iter(self) -> Self::IntoIter {
self.into_iter()
}
}
impl<T> Iterator for IntoIter<T> {
type Item = T;
fn next(&mut self) -> Option<Self::Item> {
self.list.pop_front()
}
fn size_hint(&self) -> (usize, Option<usize>) {
(self.list.len, Some(self.list.len))
}
}
impl<T> DoubleEndedIterator for IntoIter<T> {
fn next_back(&mut self) -> Option<Self::Item> {
self.list.pop_back()
}
}
impl<T> ExactSizeIterator for IntoIter<T> {
fn len(&self) -> usize {
self.list.len
}
}
Still a crapload of boiler plate, but at least it's satisfying boilerplate.
Alright, here's all of our code with all the combinatorics filled in:
#![allow(unused)] fn main() { use std::ptr::NonNull; use std::marker::PhantomData; pub struct LinkedList<T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<T>, } type Link<T> = Option<NonNull<Node<T>>>; struct Node<T> { front: Link<T>, back: Link<T>, elem: T, } pub struct Iter<'a, T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<&'a T>, } pub struct IterMut<'a, T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<&'a mut T>, } pub struct IntoIter<T> { list: LinkedList<T>, } impl<T> LinkedList<T> { pub fn new() -> Self { Self { front: None, back: None, len: 0, _boo: PhantomData, } } pub fn push_front(&mut self, elem: T) { // SAFETY: it's a linked-list, what do you want? unsafe { let new = NonNull::new_unchecked(Box::into_raw(Box::new(Node { front: None, back: None, elem, }))); if let Some(old) = self.front { // Put the new front before the old one (*old.as_ptr()).front = Some(new); (*new.as_ptr()).back = Some(old); } else { // If there's no front, then we're the empty list and need // to set the back too. self.back = Some(new); } // These things always happen! self.front = Some(new); self.len += 1; } } pub fn push_back(&mut self, elem: T) { // SAFETY: it's a linked-list, what do you want? unsafe { let new = NonNull::new_unchecked(Box::into_raw(Box::new(Node { back: None, front: None, elem, }))); if let Some(old) = self.back { // Put the new back before the old one (*old.as_ptr()).back = Some(new); (*new.as_ptr()).front = Some(old); } else { // If there's no back, then we're the empty list and need // to set the front too. self.front = Some(new); } // These things always happen! self.back = Some(new); self.len += 1; } } pub fn pop_front(&mut self) -> Option<T> { unsafe { // Only have to do stuff if there is a front node to pop. self.front.map(|node| { // Bring the Box back to life so we can move out its value and // Drop it (Box continues to magically understand this for us). let boxed_node = Box::from_raw(node.as_ptr()); let result = boxed_node.elem; // Make the next node into the new front. self.front = boxed_node.back; if let Some(new) = self.front { // Cleanup its reference to the removed node (*new.as_ptr()).front = None; } else { // If the front is now null, then this list is now empty! self.back = None; } self.len -= 1; result // Box gets implicitly freed here, knows there is no T. }) } } pub fn pop_back(&mut self) -> Option<T> { unsafe { // Only have to do stuff if there is a back node to pop. self.back.map(|node| { // Bring the Box front to life so we can move out its value and // Drop it (Box continues to magically understand this for us). let boxed_node = Box::from_raw(node.as_ptr()); let result = boxed_node.elem; // Make the next node into the new back. self.back = boxed_node.front; if let Some(new) = self.back { // Cleanup its reference to the removed node (*new.as_ptr()).back = None; } else { // If the back is now null, then this list is now empty! self.front = None; } self.len -= 1; result // Box gets implicitly freed here, knows there is no T. }) } } pub fn front(&self) -> Option<&T> { unsafe { self.front.map(|node| &(*node.as_ptr()).elem) } } pub fn front_mut(&mut self) -> Option<&mut T> { unsafe { self.front.map(|node| &mut (*node.as_ptr()).elem) } } pub fn back(&self) -> Option<&T> { unsafe { self.back.map(|node| &(*node.as_ptr()).elem) } } pub fn back_mut(&mut self) -> Option<&mut T> { unsafe { self.back.map(|node| &mut (*node.as_ptr()).elem) } } pub fn len(&self) -> usize { self.len } pub fn iter(&self) -> Iter<T> { Iter { front: self.front, back: self.back, len: self.len, _boo: PhantomData, } } pub fn iter_mut(&mut self) -> IterMut<T> { IterMut { front: self.front, back: self.back, len: self.len, _boo: PhantomData, } } pub fn into_iter(self) -> IntoIter<T> { IntoIter { list: self } } } impl<T> Drop for LinkedList<T> { fn drop(&mut self) { // Pop until we have to stop while let Some(_) = self.pop_front() { } } } impl<'a, T> IntoIterator for &'a LinkedList<T> { type IntoIter = Iter<'a, T>; type Item = &'a T; fn into_iter(self) -> Self::IntoIter { self.iter() } } impl<'a, T> Iterator for Iter<'a, T> { type Item = &'a T; fn next(&mut self) -> Option<Self::Item> { // While self.front == self.back is a tempting condition to check here, // it won't do the right for yielding the last element! That sort of // thing only works for arrays because of "one-past-the-end" pointers. if self.len > 0 { // We could unwrap front, but this is safer and easier self.front.map(|node| unsafe { self.len -= 1; self.front = (*node.as_ptr()).back; &(*node.as_ptr()).elem }) } else { None } } fn size_hint(&self) -> (usize, Option<usize>) { (self.len, Some(self.len)) } } impl<'a, T> DoubleEndedIterator for Iter<'a, T> { fn next_back(&mut self) -> Option<Self::Item> { if self.len > 0 { self.back.map(|node| unsafe { self.len -= 1; self.back = (*node.as_ptr()).front; &(*node.as_ptr()).elem }) } else { None } } } impl<'a, T> ExactSizeIterator for Iter<'a, T> { fn len(&self) -> usize { self.len } } impl<'a, T> IntoIterator for &'a mut LinkedList<T> { type IntoIter = IterMut<'a, T>; type Item = &'a mut T; fn into_iter(self) -> Self::IntoIter { self.iter_mut() } } impl<'a, T> Iterator for IterMut<'a, T> { type Item = &'a mut T; fn next(&mut self) -> Option<Self::Item> { // While self.front == self.back is a tempting condition to check here, // it won't do the right for yielding the last element! That sort of // thing only works for arrays because of "one-past-the-end" pointers. if self.len > 0 { // We could unwrap front, but this is safer and easier self.front.map(|node| unsafe { self.len -= 1; self.front = (*node.as_ptr()).back; &mut (*node.as_ptr()).elem }) } else { None } } fn size_hint(&self) -> (usize, Option<usize>) { (self.len, Some(self.len)) } } impl<'a, T> DoubleEndedIterator for IterMut<'a, T> { fn next_back(&mut self) -> Option<Self::Item> { if self.len > 0 { self.back.map(|node| unsafe { self.len -= 1; self.back = (*node.as_ptr()).front; &mut (*node.as_ptr()).elem }) } else { None } } } impl<'a, T> ExactSizeIterator for IterMut<'a, T> { fn len(&self) -> usize { self.len } } impl<T> IntoIterator for LinkedList<T> { type IntoIter = IntoIter<T>; type Item = T; fn into_iter(self) -> Self::IntoIter { self.into_iter() } } impl<T> Iterator for IntoIter<T> { type Item = T; fn next(&mut self) -> Option<Self::Item> { self.list.pop_front() } fn size_hint(&self) -> (usize, Option<usize>) { (self.list.len, Some(self.list.len)) } } impl<T> DoubleEndedIterator for IntoIter<T> { fn next_back(&mut self) -> Option<Self::Item> { self.list.pop_back() } } impl<T> ExactSizeIterator for IntoIter<T> { fn len(&self) -> usize { self.list.len } } #[cfg(test)] mod test { use super::LinkedList; #[test] fn test_basic_front() { let mut list = LinkedList::new(); // Try to break an empty list assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); // Try to break a one item list list.push_front(10); assert_eq!(list.len(), 1); assert_eq!(list.pop_front(), Some(10)); assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); // Mess around list.push_front(10); assert_eq!(list.len(), 1); list.push_front(20); assert_eq!(list.len(), 2); list.push_front(30); assert_eq!(list.len(), 3); assert_eq!(list.pop_front(), Some(30)); assert_eq!(list.len(), 2); list.push_front(40); assert_eq!(list.len(), 3); assert_eq!(list.pop_front(), Some(40)); assert_eq!(list.len(), 2); assert_eq!(list.pop_front(), Some(20)); assert_eq!(list.len(), 1); assert_eq!(list.pop_front(), Some(10)); assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); } } }
Filling In Random Bits
Hey you said you wanted to be production-quality, didn't you?
Here's some more random gunk to toss in to be a "good" collection:
impl<T> LinkedList<T> {
pub fn is_empty(&self) -> bool {
self.len == 0
}
pub fn clear(&mut self) {
// Oh look it's drop again
while let Some(_) = self.pop_front() { }
}
}
And now we've got a bunch of traits to implement that everyone expects:
impl<T> Default for LinkedList<T> {
fn default() -> Self {
Self::new()
}
}
impl<T: Clone> Clone for LinkedList<T> {
fn clone(&self) -> Self {
let mut new_list = Self::new();
for item in self {
new_list.push_back(item.clone());
}
new_list
}
}
impl<T> Extend<T> for LinkedList<T> {
fn extend<I: IntoIterator<Item = T>>(&mut self, iter: I) {
for item in iter {
self.push_back(item);
}
}
}
impl<T> FromIterator<T> for LinkedList<T> {
fn from_iter<I: IntoIterator<Item = T>>(iter: I) -> Self {
let mut list = Self::new();
list.extend(iter);
list
}
}
impl<T: Debug> Debug for LinkedList<T> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_list().entries(self).finish()
}
}
impl<T: PartialEq> PartialEq for LinkedList<T> {
fn eq(&self, other: &Self) -> bool {
self.len() == other.len() && self.iter().eq(other)
}
fn ne(&self, other: &Self) -> bool {
self.len() != other.len() || self.iter().ne(other)
}
}
impl<T: Eq> Eq for LinkedList<T> { }
impl<T: PartialOrd> PartialOrd for LinkedList<T> {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
self.iter().partial_cmp(other)
}
}
impl<T: Ord> Ord for LinkedList<T> {
fn cmp(&self, other: &Self) -> Ordering {
self.iter().cmp(other)
}
}
impl<T: Hash> Hash for LinkedList<T> {
fn hash<H: Hasher>(&self, state: &mut H) {
self.len().hash(state);
for item in self {
item.hash(state);
}
}
}
I definitely wrote all of these from scratch, and didn't just copy the std impls. Because they're so interesting, and I definitely remember the subtleties of manually implementing Hash. Yeah, that's something I think about All The Time...
Ok there's actually a few things worth noting here.
First, a nasty namespace clash. For whatever reason std now has macros named Hash and Debug, and so if you don't have the traits imported, you'll get really cryptic errors about macros instead of the proper "missing trait".
The other intersting thing to talk about is Hash itself. Do you see how we hash in len? That's actually really important! If collections don't hash in lengths, they can accidentally make themselves vulnerable to prefix collisions. For instance, what distinguishes ["he", "llo"] from ["hello"]? If no one is hashing lengths or some other "separator", nothing! Making it too easy for hash collisions to accidentally or maliciously happen can result in serious sadness, so just do it!
Alright, here's our current code:
#![allow(unused)] fn main() { use std::cmp::Ordering; use std::fmt::{self, Debug}; use std::hash::{Hash, Hasher}; use std::iter::FromIterator; use std::ptr::NonNull; use std::marker::PhantomData; pub struct LinkedList<T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<T>, } type Link<T> = Option<NonNull<Node<T>>>; struct Node<T> { front: Link<T>, back: Link<T>, elem: T, } pub struct Iter<'a, T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<&'a T>, } pub struct IterMut<'a, T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<&'a mut T>, } pub struct IntoIter<T> { list: LinkedList<T>, } impl<T> LinkedList<T> { pub fn new() -> Self { Self { front: None, back: None, len: 0, _boo: PhantomData, } } pub fn push_front(&mut self, elem: T) { // SAFETY: it's a linked-list, what do you want? unsafe { let new = NonNull::new_unchecked(Box::into_raw(Box::new(Node { front: None, back: None, elem, }))); if let Some(old) = self.front { // Put the new front before the old one (*old.as_ptr()).front = Some(new); (*new.as_ptr()).back = Some(old); } else { // If there's no front, then we're the empty list and need // to set the back too. self.back = Some(new); } // These things always happen! self.front = Some(new); self.len += 1; } } pub fn push_back(&mut self, elem: T) { // SAFETY: it's a linked-list, what do you want? unsafe { let new = NonNull::new_unchecked(Box::into_raw(Box::new(Node { back: None, front: None, elem, }))); if let Some(old) = self.back { // Put the new back before the old one (*old.as_ptr()).back = Some(new); (*new.as_ptr()).front = Some(old); } else { // If there's no back, then we're the empty list and need // to set the front too. self.front = Some(new); } // These things always happen! self.back = Some(new); self.len += 1; } } pub fn pop_front(&mut self) -> Option<T> { unsafe { // Only have to do stuff if there is a front node to pop. self.front.map(|node| { // Bring the Box back to life so we can move out its value and // Drop it (Box continues to magically understand this for us). let boxed_node = Box::from_raw(node.as_ptr()); let result = boxed_node.elem; // Make the next node into the new front. self.front = boxed_node.back; if let Some(new) = self.front { // Cleanup its reference to the removed node (*new.as_ptr()).front = None; } else { // If the front is now null, then this list is now empty! self.back = None; } self.len -= 1; result // Box gets implicitly freed here, knows there is no T. }) } } pub fn pop_back(&mut self) -> Option<T> { unsafe { // Only have to do stuff if there is a back node to pop. self.back.map(|node| { // Bring the Box front to life so we can move out its value and // Drop it (Box continues to magically understand this for us). let boxed_node = Box::from_raw(node.as_ptr()); let result = boxed_node.elem; // Make the next node into the new back. self.back = boxed_node.front; if let Some(new) = self.back { // Cleanup its reference to the removed node (*new.as_ptr()).back = None; } else { // If the back is now null, then this list is now empty! self.front = None; } self.len -= 1; result // Box gets implicitly freed here, knows there is no T. }) } } pub fn front(&self) -> Option<&T> { unsafe { self.front.map(|node| &(*node.as_ptr()).elem) } } pub fn front_mut(&mut self) -> Option<&mut T> { unsafe { self.front.map(|node| &mut (*node.as_ptr()).elem) } } pub fn back(&self) -> Option<&T> { unsafe { self.back.map(|node| &(*node.as_ptr()).elem) } } pub fn back_mut(&mut self) -> Option<&mut T> { unsafe { self.back.map(|node| &mut (*node.as_ptr()).elem) } } pub fn len(&self) -> usize { self.len } pub fn is_empty(&self) -> bool { self.len == 0 } pub fn clear(&mut self) { // Oh look it's drop again while let Some(_) = self.pop_front() { } } pub fn iter(&self) -> Iter<T> { Iter { front: self.front, back: self.back, len: self.len, _boo: PhantomData, } } pub fn iter_mut(&mut self) -> IterMut<T> { IterMut { front: self.front, back: self.back, len: self.len, _boo: PhantomData, } } pub fn into_iter(self) -> IntoIter<T> { IntoIter { list: self } } } impl<T> Drop for LinkedList<T> { fn drop(&mut self) { // Pop until we have to stop while let Some(_) = self.pop_front() { } } } impl<T> Default for LinkedList<T> { fn default() -> Self { Self::new() } } impl<T: Clone> Clone for LinkedList<T> { fn clone(&self) -> Self { let mut new_list = Self::new(); for item in self { new_list.push_back(item.clone()); } new_list } } impl<T> Extend<T> for LinkedList<T> { fn extend<I: IntoIterator<Item = T>>(&mut self, iter: I) { for item in iter { self.push_back(item); } } } impl<T> FromIterator<T> for LinkedList<T> { fn from_iter<I: IntoIterator<Item = T>>(iter: I) -> Self { let mut list = Self::new(); list.extend(iter); list } } impl<T: Debug> Debug for LinkedList<T> { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.debug_list().entries(self).finish() } } impl<T: PartialEq> PartialEq for LinkedList<T> { fn eq(&self, other: &Self) -> bool { self.len() == other.len() && self.iter().eq(other) } fn ne(&self, other: &Self) -> bool { self.len() != other.len() || self.iter().ne(other) } } impl<T: Eq> Eq for LinkedList<T> { } impl<T: PartialOrd> PartialOrd for LinkedList<T> { fn partial_cmp(&self, other: &Self) -> Option<Ordering> { self.iter().partial_cmp(other) } } impl<T: Ord> Ord for LinkedList<T> { fn cmp(&self, other: &Self) -> Ordering { self.iter().cmp(other) } } impl<T: Hash> Hash for LinkedList<T> { fn hash<H: Hasher>(&self, state: &mut H) { self.len().hash(state); for item in self { item.hash(state); } } } impl<'a, T> IntoIterator for &'a LinkedList<T> { type IntoIter = Iter<'a, T>; type Item = &'a T; fn into_iter(self) -> Self::IntoIter { self.iter() } } impl<'a, T> Iterator for Iter<'a, T> { type Item = &'a T; fn next(&mut self) -> Option<Self::Item> { // While self.front == self.back is a tempting condition to check here, // it won't do the right for yielding the last element! That sort of // thing only works for arrays because of "one-past-the-end" pointers. if self.len > 0 { // We could unwrap front, but this is safer and easier self.front.map(|node| unsafe { self.len -= 1; self.front = (*node.as_ptr()).back; &(*node.as_ptr()).elem }) } else { None } } fn size_hint(&self) -> (usize, Option<usize>) { (self.len, Some(self.len)) } } impl<'a, T> DoubleEndedIterator for Iter<'a, T> { fn next_back(&mut self) -> Option<Self::Item> { if self.len > 0 { self.back.map(|node| unsafe { self.len -= 1; self.back = (*node.as_ptr()).front; &(*node.as_ptr()).elem }) } else { None } } } impl<'a, T> ExactSizeIterator for Iter<'a, T> { fn len(&self) -> usize { self.len } } impl<'a, T> IntoIterator for &'a mut LinkedList<T> { type IntoIter = IterMut<'a, T>; type Item = &'a mut T; fn into_iter(self) -> Self::IntoIter { self.iter_mut() } } impl<'a, T> Iterator for IterMut<'a, T> { type Item = &'a mut T; fn next(&mut self) -> Option<Self::Item> { // While self.front == self.back is a tempting condition to check here, // it won't do the right for yielding the last element! That sort of // thing only works for arrays because of "one-past-the-end" pointers. if self.len > 0 { // We could unwrap front, but this is safer and easier self.front.map(|node| unsafe { self.len -= 1; self.front = (*node.as_ptr()).back; &mut (*node.as_ptr()).elem }) } else { None } } fn size_hint(&self) -> (usize, Option<usize>) { (self.len, Some(self.len)) } } impl<'a, T> DoubleEndedIterator for IterMut<'a, T> { fn next_back(&mut self) -> Option<Self::Item> { if self.len > 0 { self.back.map(|node| unsafe { self.len -= 1; self.back = (*node.as_ptr()).front; &mut (*node.as_ptr()).elem }) } else { None } } } impl<'a, T> ExactSizeIterator for IterMut<'a, T> { fn len(&self) -> usize { self.len } } impl<T> IntoIterator for LinkedList<T> { type IntoIter = IntoIter<T>; type Item = T; fn into_iter(self) -> Self::IntoIter { self.into_iter() } } impl<T> Iterator for IntoIter<T> { type Item = T; fn next(&mut self) -> Option<Self::Item> { self.list.pop_front() } fn size_hint(&self) -> (usize, Option<usize>) { (self.list.len, Some(self.list.len)) } } impl<T> DoubleEndedIterator for IntoIter<T> { fn next_back(&mut self) -> Option<Self::Item> { self.list.pop_back() } } impl<T> ExactSizeIterator for IntoIter<T> { fn len(&self) -> usize { self.list.len } } #[cfg(test)] mod test { use super::LinkedList; #[test] fn test_basic_front() { let mut list = LinkedList::new(); // Try to break an empty list assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); // Try to break a one item list list.push_front(10); assert_eq!(list.len(), 1); assert_eq!(list.pop_front(), Some(10)); assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); // Mess around list.push_front(10); assert_eq!(list.len(), 1); list.push_front(20); assert_eq!(list.len(), 2); list.push_front(30); assert_eq!(list.len(), 3); assert_eq!(list.pop_front(), Some(30)); assert_eq!(list.len(), 2); list.push_front(40); assert_eq!(list.len(), 3); assert_eq!(list.pop_front(), Some(40)); assert_eq!(list.len(), 2); assert_eq!(list.pop_front(), Some(20)); assert_eq!(list.len(), 1); assert_eq!(list.pop_front(), Some(10)); assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); } } }
Testing
Alright I put off testing for a while because well, we both know that we're now masters of Rust and we don't make mistakes anymore! Also just, this is a rewrite of an old crate, so I just have all the tests already. They're tests, you've seen tests a lot already. Here they are:
#[cfg(test)]
mod test {
use super::LinkedList;
fn generate_test() -> LinkedList<i32> {
list_from(&[0, 1, 2, 3, 4, 5, 6])
}
fn list_from<T: Clone>(v: &[T]) -> LinkedList<T> {
v.iter().map(|x| (*x).clone()).collect()
}
#[test]
fn test_basic_front() {
let mut list = LinkedList::new();
// Try to break an empty list
assert_eq!(list.len(), 0);
assert_eq!(list.pop_front(), None);
assert_eq!(list.len(), 0);
// Try to break a one item list
list.push_front(10);
assert_eq!(list.len(), 1);
assert_eq!(list.pop_front(), Some(10));
assert_eq!(list.len(), 0);
assert_eq!(list.pop_front(), None);
assert_eq!(list.len(), 0);
// Mess around
list.push_front(10);
assert_eq!(list.len(), 1);
list.push_front(20);
assert_eq!(list.len(), 2);
list.push_front(30);
assert_eq!(list.len(), 3);
assert_eq!(list.pop_front(), Some(30));
assert_eq!(list.len(), 2);
list.push_front(40);
assert_eq!(list.len(), 3);
assert_eq!(list.pop_front(), Some(40));
assert_eq!(list.len(), 2);
assert_eq!(list.pop_front(), Some(20));
assert_eq!(list.len(), 1);
assert_eq!(list.pop_front(), Some(10));
assert_eq!(list.len(), 0);
assert_eq!(list.pop_front(), None);
assert_eq!(list.len(), 0);
assert_eq!(list.pop_front(), None);
assert_eq!(list.len(), 0);
}
#[test]
fn test_basic() {
let mut m = LinkedList::new();
assert_eq!(m.pop_front(), None);
assert_eq!(m.pop_back(), None);
assert_eq!(m.pop_front(), None);
m.push_front(1);
assert_eq!(m.pop_front(), Some(1));
m.push_back(2);
m.push_back(3);
assert_eq!(m.len(), 2);
assert_eq!(m.pop_front(), Some(2));
assert_eq!(m.pop_front(), Some(3));
assert_eq!(m.len(), 0);
assert_eq!(m.pop_front(), None);
m.push_back(1);
m.push_back(3);
m.push_back(5);
m.push_back(7);
assert_eq!(m.pop_front(), Some(1));
let mut n = LinkedList::new();
n.push_front(2);
n.push_front(3);
{
assert_eq!(n.front().unwrap(), &3);
let x = n.front_mut().unwrap();
assert_eq!(*x, 3);
*x = 0;
}
{
assert_eq!(n.back().unwrap(), &2);
let y = n.back_mut().unwrap();
assert_eq!(*y, 2);
*y = 1;
}
assert_eq!(n.pop_front(), Some(0));
assert_eq!(n.pop_front(), Some(1));
}
#[test]
fn test_iterator() {
let m = generate_test();
for (i, elt) in m.iter().enumerate() {
assert_eq!(i as i32, *elt);
}
let mut n = LinkedList::new();
assert_eq!(n.iter().next(), None);
n.push_front(4);
let mut it = n.iter();
assert_eq!(it.size_hint(), (1, Some(1)));
assert_eq!(it.next().unwrap(), &4);
assert_eq!(it.size_hint(), (0, Some(0)));
assert_eq!(it.next(), None);
}
#[test]
fn test_iterator_double_end() {
let mut n = LinkedList::new();
assert_eq!(n.iter().next(), None);
n.push_front(4);
n.push_front(5);
n.push_front(6);
let mut it = n.iter();
assert_eq!(it.size_hint(), (3, Some(3)));
assert_eq!(it.next().unwrap(), &6);
assert_eq!(it.size_hint(), (2, Some(2)));
assert_eq!(it.next_back().unwrap(), &4);
assert_eq!(it.size_hint(), (1, Some(1)));
assert_eq!(it.next_back().unwrap(), &5);
assert_eq!(it.next_back(), None);
assert_eq!(it.next(), None);
}
#[test]
fn test_rev_iter() {
let m = generate_test();
for (i, elt) in m.iter().rev().enumerate() {
assert_eq!(6 - i as i32, *elt);
}
let mut n = LinkedList::new();
assert_eq!(n.iter().rev().next(), None);
n.push_front(4);
let mut it = n.iter().rev();
assert_eq!(it.size_hint(), (1, Some(1)));
assert_eq!(it.next().unwrap(), &4);
assert_eq!(it.size_hint(), (0, Some(0)));
assert_eq!(it.next(), None);
}
#[test]
fn test_mut_iter() {
let mut m = generate_test();
let mut len = m.len();
for (i, elt) in m.iter_mut().enumerate() {
assert_eq!(i as i32, *elt);
len -= 1;
}
assert_eq!(len, 0);
let mut n = LinkedList::new();
assert!(n.iter_mut().next().is_none());
n.push_front(4);
n.push_back(5);
let mut it = n.iter_mut();
assert_eq!(it.size_hint(), (2, Some(2)));
assert!(it.next().is_some());
assert!(it.next().is_some());
assert_eq!(it.size_hint(), (0, Some(0)));
assert!(it.next().is_none());
}
#[test]
fn test_iterator_mut_double_end() {
let mut n = LinkedList::new();
assert!(n.iter_mut().next_back().is_none());
n.push_front(4);
n.push_front(5);
n.push_front(6);
let mut it = n.iter_mut();
assert_eq!(it.size_hint(), (3, Some(3)));
assert_eq!(*it.next().unwrap(), 6);
assert_eq!(it.size_hint(), (2, Some(2)));
assert_eq!(*it.next_back().unwrap(), 4);
assert_eq!(it.size_hint(), (1, Some(1)));
assert_eq!(*it.next_back().unwrap(), 5);
assert!(it.next_back().is_none());
assert!(it.next().is_none());
}
#[test]
fn test_eq() {
let mut n: LinkedList<u8> = list_from(&[]);
let mut m = list_from(&[]);
assert!(n == m);
n.push_front(1);
assert!(n != m);
m.push_back(1);
assert!(n == m);
let n = list_from(&[2, 3, 4]);
let m = list_from(&[1, 2, 3]);
assert!(n != m);
}
#[test]
fn test_ord() {
let n = list_from(&[]);
let m = list_from(&[1, 2, 3]);
assert!(n < m);
assert!(m > n);
assert!(n <= n);
assert!(n >= n);
}
#[test]
fn test_ord_nan() {
let nan = 0.0f64 / 0.0;
let n = list_from(&[nan]);
let m = list_from(&[nan]);
assert!(!(n < m));
assert!(!(n > m));
assert!(!(n <= m));
assert!(!(n >= m));
let n = list_from(&[nan]);
let one = list_from(&[1.0f64]);
assert!(!(n < one));
assert!(!(n > one));
assert!(!(n <= one));
assert!(!(n >= one));
let u = list_from(&[1.0f64, 2.0, nan]);
let v = list_from(&[1.0f64, 2.0, 3.0]);
assert!(!(u < v));
assert!(!(u > v));
assert!(!(u <= v));
assert!(!(u >= v));
let s = list_from(&[1.0f64, 2.0, 4.0, 2.0]);
let t = list_from(&[1.0f64, 2.0, 3.0, 2.0]);
assert!(!(s < t));
assert!(s > one);
assert!(!(s <= one));
assert!(s >= one);
}
#[test]
fn test_debug() {
let list: LinkedList<i32> = (0..10).collect();
assert_eq!(format!("{:?}", list), "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]");
let list: LinkedList<&str> = vec!["just", "one", "test", "more"]
.iter().copied()
.collect();
assert_eq!(format!("{:?}", list), r#"["just", "one", "test", "more"]"#);
}
#[test]
fn test_hashmap() {
// Check that HashMap works with this as a key
let list1: LinkedList<i32> = (0..10).collect();
let list2: LinkedList<i32> = (1..11).collect();
let mut map = std::collections::HashMap::new();
assert_eq!(map.insert(list1.clone(), "list1"), None);
assert_eq!(map.insert(list2.clone(), "list2"), None);
assert_eq!(map.len(), 2);
assert_eq!(map.get(&list1), Some(&"list1"));
assert_eq!(map.get(&list2), Some(&"list2"));
assert_eq!(map.remove(&list1), Some("list1"));
assert_eq!(map.remove(&list2), Some("list2"));
assert!(map.is_empty());
}
}
And now for the moments of truth:
cargo test
Finished test [unoptimized + debuginfo] target(s) in 0.00s
Running unittests src\lib.rs
running 12 tests
test test::test_basic ... ok
test test::test_basic_front ... ok
test test::test_eq ... ok
test test::test_iterator ... ok
test test::test_iterator_mut_double_end ... ok
test test::test_ord_nan ... ok
test test::test_iterator_double_end ... ok
test test::test_mut_iter ... ok
test test::test_rev_iter ... ok
test test::test_hashmap ... ok
test test::test_ord ... ok
test test::test_debug ... ok
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
$env:MIRIFLAGS="-Zmiri-tag-raw-pointers"
cargo miri test
Compiling linked-list v0.0.3
Finished test [unoptimized + debuginfo] target(s) in 0.35s
Running unittests src\lib.rs
running 12 tests
test test::test_basic ... ok
test test::test_basic_front ... ok
test test::test_debug ... ok
test test::test_eq ... ok
test test::test_hashmap ... ok
test test::test_iterator ... ok
test test::test_iterator_double_end ... ok
test test::test_iterator_mut_double_end ... ok
test test::test_mut_iter ... ok
test test::test_ord ... ok
test test::test_ord_nan ... ok
test test::test_rev_iter ... ok
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
😭
We did it, we actually didn't screw up. This isn't a trick! All of our practice and training was finally worth it, we've finally written good code!!!
Now with all that gunk out of the way, we can get back to the Interesting Stuff!
Send, Sync, and Compile Tests
Ok actually we do have one more pair of traits to think about, but they're special. We have to deal with Rust's Holy Roman Empire: The Unsafe Opt-In Built-In Traits (OIBITs): Send and Sync, which are in fact opt-out and built-out (1 out of 3 is pretty good!).
Like Copy, these traits have absolutely no code associated with them, and are just markers that your type has a particular property. Send says that your type is safe to send to another thread. Sync says your type is safe to share between threads (&Self: Send).
The same argument for LinkedList being covariant applies here: generally normal collections which don't use fancy interior mutability tricks are safe to make Send and Sync.
But I said they're opt out. So actually, are we already? How would we know?
Let's add some new magic to our code: random private garbage that won't compile unless our types have the properties we expect:
#[allow(dead_code)]
fn assert_properties() {
fn is_send<T: Send>() {}
fn is_sync<T: Sync>() {}
is_send::<LinkedList<i32>>();
is_sync::<LinkedList<i32>>();
is_send::<IntoIter<i32>>();
is_sync::<IntoIter<i32>>();
is_send::<Iter<i32>>();
is_sync::<Iter<i32>>();
is_send::<IterMut<i32>>();
is_sync::<IterMut<i32>>();
is_send::<Cursor<i32>>();
is_sync::<Cursor<i32>>();
fn linked_list_covariant<'a, T>(x: LinkedList<&'static T>) -> LinkedList<&'a T> { x }
fn iter_covariant<'i, 'a, T>(x: Iter<'i, &'static T>) -> Iter<'i, &'a T> { x }
fn into_iter_covariant<'a, T>(x: IntoIter<&'static T>) -> IntoIter<&'a T> { x }
}
cargo build
Compiling linked-list v0.0.3
error[E0277]: `NonNull<Node<i32>>` cannot be sent between threads safely
--> src\lib.rs:433:5
|
433 | is_send::<LinkedList<i32>>();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ `NonNull<Node<i32>>` cannot be sent between threads safely
|
= help: within `LinkedList<i32>`, the trait `Send` is not implemented for `NonNull<Node<i32>>`
= note: required because it appears within the type `Option<NonNull<Node<i32>>>`
note: required because it appears within the type `LinkedList<i32>`
--> src\lib.rs:8:12
|
8 | pub struct LinkedList<T> {
| ^^^^^^^^^^
note: required by a bound in `is_send`
--> src\lib.rs:430:19
|
430 | fn is_send<T: Send>() {}
| ^^^^ required by this bound in `is_send`
<a million more errors>
Oh geez, what gives! I had that great Holy Roman Empire joke!
Well, I lied to you when I said raw pointers have only one safety guard: this is the other. *const AND *mut explicitly opt out of Send and Sync to be safe, so we do actually have to opt back in:
unsafe impl<T: Send> Send for LinkedList<T> {}
unsafe impl<T: Sync> Sync for LinkedList<T> {}
unsafe impl<'a, T: Send> Send for Iter<'a, T> {}
unsafe impl<'a, T: Sync> Sync for Iter<'a, T> {}
unsafe impl<'a, T: Send> Send for IterMut<'a, T> {}
unsafe impl<'a, T: Sync> Sync for IterMut<'a, T> {}
Note that we have to write unsafe impl here: these are unsafe traits! Unsafe code (like concurrency libraries) gets to rely on us only implementing these traits correctly! Since there's no actual code, the guarantee we're making is just that, yes, we are indeed safe to Send or Share between threads!
Don't just slap these on lightly, but I am a Certified Professional here to say: yep there's are totally fine. Note how we don't need to implement Send and Sync for IntoIter: it just contains LinkedList, so it auto-derives Send and Sync — I told you they were actually opt out! (You opt out with the hillarious syntax of impl !Send for MyType {}.)
cargo build
Compiling linked-list v0.0.3
Finished dev [unoptimized + debuginfo] target(s) in 0.18s
Ok nice!
...Wait, actually it would be really dangerous if stuff that shouldn't be these things wasn't. In particular, IterMut definitely shouldn't be covariant, because it's "like" &mut T. But how can we check that?
With Magic! Well, actually, with rustdoc! Ok well we don't have to use rustdoc for this, but it's the funniest way to do it. See, if you write a doccomment and include a code block, then rustdoc will try to compile and run it, so we can use that to make fresh anonymous "programs" that don't affect the main one:
/// ```
/// use linked_list::IterMut;
///
/// fn iter_mut_covariant<'i, 'a, T>(x: IterMut<'i, &'static T>) -> IterMut<'i, &'a T> { x }
/// ```
fn iter_mut_invariant() {}
cargo test
...
Doc-tests linked-list
running 1 test
test src\lib.rs - assert_properties::iter_mut_invariant (line 458) ... FAILED
failures:
---- src\lib.rs - assert_properties::iter_mut_invariant (line 458) stdout ----
error[E0308]: mismatched types
--> src\lib.rs:461:86
|
6 | fn iter_mut_covariant<'i, 'a, T>(x: IterMut<'i, &'static T>) -> IterMut<'i, &'a T> { x }
| ^ lifetime mismatch
|
= note: expected struct `linked_list::IterMut<'_, &'a T>`
found struct `linked_list::IterMut<'_, &'static T>`
Ok cool, we've proved it's invariant, but uh, now our tests fail. No worries, rustdoc lets you say that's expected by annotating the fence with compile_fail!
(Actually we only proved it's "not covariant" but honestly if you manage to make a type "accidentaly and incorrectly contravariant" then, congrats?)
/// ```compile_fail
/// use linked_list::IterMut;
///
/// fn iter_mut_covariant<'i, 'a, T>(x: IterMut<'i, &'static T>) -> IterMut<'i, &'a T> { x }
/// ```
fn iter_mut_invariant() {}
cargo test
Compiling linked-list v0.0.3
Finished test [unoptimized + debuginfo] target(s) in 0.49s
Running unittests src\lib.rs
...
Doc-tests linked-list
running 1 test
test src\lib.rs - assert_properties::iter_mut_invariant (line 458) - compile fail ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s
Yay! I recommend always making the test without compile_fail so that you can confirm that it fails to compile for the right reason. For instance, that test will also fail (and therefore pass) if you forget the use, which, is not what we want! While it's conceptually appealing to be able to "require" a specific error from the compiler, this would be an absolute nightmare that would effectively make it a breaking change for the compiler to produce better errors. We want the compiler to get better, so, no you don't get to have that.
(Oh wait, we can actually just specify the error code we want next to the compile_fail but this only works on nightly and is a bad idea to rely on for the reasons state above. It will be silently ignored on not-nightly.)
/// ```compile_fail,E0308
/// use linked_list::IterMut;
///
/// fn iter_mut_covariant<'i, 'a, T>(x: IterMut<'i, &'static T>) -> IterMut<'i, &'a T> { x }
/// ```
fn iter_mut_invariant() {}
...also, did you notice the part where we actually made IterMut invariant? It was easy to miss, since I "just" copy-pasted Iter and dumped it at the end. It's the last line here:
pub struct IterMut<'a, T> {
front: Link<T>,
back: Link<T>,
len: usize,
_boo: PhantomData<&'a mut T>,
}
Let's try removing that PhantomData:
cargo build
Compiling linked-list v0.0.3 (C:\Users\ninte\dev\contain\linked-list)
error[E0392]: parameter `'a` is never used
--> src\lib.rs:30:20
|
30 | pub struct IterMut<'a, T> {
| ^^ unused parameter
|
= help: consider removing `'a`, referring to it in a field, or using a marker such as `PhantomData`
Ha! The compiler has our back and won't just let us not use the lifetime. Let's try using the wrong example instead:
_boo: PhantomData<&'a T>,
cargo build
Compiling linked-list v0.0.3 (C:\Users\ninte\dev\contain\linked-list)
Finished dev [unoptimized + debuginfo] target(s) in 0.17s
It builds! Do our tests catch a problem now?
cargo test
...
Doc-tests linked-list
running 1 test
test src\lib.rs - assert_properties::iter_mut_invariant (line 458) - compile fail ... FAILED
failures:
---- src\lib.rs - assert_properties::iter_mut_invariant (line 458) stdout ----
Test compiled successfully, but it's marked `compile_fail`.
failures:
src\lib.rs - assert_properties::iter_mut_invariant (line 458)
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s
Eyyy!!! The system works! I love having tests that actually do their job, so that I don't have to be quite so horrified of looming mistakes!
An Introduction To Cursors
OK!!! We now have a LinkedList that's on par with std's 1.0 implementation! Which of course means that our LinkedList is still completely useless. We've taken the enormous performance penalty of implementing a Deque as a linked list, and we don't have any of the APIs that make it actually useful.
Here's how we do against the "killer apps" of linked lists:
- 🚫 Getting to do weird intrusive stuff
- 🚫 Getting to do weird lockfree stuff
- 🚫 Getting to store Dynamically Sized Types
- 🌟 O(1) push/pop without amortization (if you are willing to believe that malloc is O(1))
- 🚫 O(1) list splitting
- 🚫 O(1) list splicing
Well... 1 out of 6 is... better than nothing! Do you see why I wanted to rip this thing out of std?
We're not going to make our list support "weird" stuff, because that's all adhoc and domain-specific. But the splitting and splicing thing, now that's something we can do!
But here's the problem: actually reaching the kth element in a LinkedList takes O(k) time, so how can we possibly do arbitrary splits and merges in O(1)? Well, the trick is that you don't have an API like split_at(index) -- you make a system where the user can statefully iterate to a position in the list and make O(1) modifications at that point!
Hey, we already have iterators! Can we use them for this? Kind of... but one of their super-powers gets in the way. You may recall that the way that we write out the lifetimes for by-ref iterators means that the references they return aren't tied to the iterator. This lets us repeatedly call next and hold onto the elements:
let mut list = ...;
let iter = list.iter_mut();
let elem1 = list.next();
let elem2 = list.next();
if elem1 == elem2 { ... }
If the returned references borrowed the iterator, then this code wouldn't work at all. The compiler would just complain about the second call to next! This flexibility is great, but it puts some implicit constraints on us:
-
By-Mutable-Ref Iterators can never go backwards and yield an element again, because the user would be able to get two
&mut's to the same element, breaking fundamental rules of the language. -
By-Ref Iterators can't have extra methods which could possibly modify the underlying collection in a way that would invalidate any reference that has already been yielded.
Unfortunately, both of these things are exactly what we want our LinkedList API to do! So we can't just use iterators, we need something new: Cursors.
Cursors are exactly like the little blinking | you get when you're editing some text on a computer. It's a position in a sequence (the text) that you can move around (with the arrow keys), and whenever you type the edits happen at that point.
See if I just
press
enter
the whole
text
gets broken in half.
Sorry you're standing behind me and watching me type this right? So that totally makes sense, right? Right.
Now if you've ever had the misfortune of having a keyboard with an "insert" key and actually pressed it, you know that there's actually technically two interpretations of cursors: they can either lie between elements (characters) or on elements. I'm pretty sure no one has ever pressed "insert" on purpose in their life, and that it exists purely as a Suffering Button, so it's pretty obvious which one is Better and Right: cursors go between elements!
Pretty rock-solid logic right there, I don't think anyone can disagree with me.
Sorry what? There was an RFC in 2018 to add Cursors to Rust's LinkedList?
With a Cursor one can seek back and forth through a list and get the current element. With a CursorMut One can seek back and forth and get mutable references to elements, and it can insert and delete elements before and behind the current element (along with performing several list operations such as splitting and splicing).
Current element? This cursor is on elements, not between them! I can't believe they didn't accept my totally rock-solid argument! So yeah you can just go use the Cursor in std... wait, it's 2022, and Rust 1.60 still has Cursor marked as unstable?
Hey wait:
Cursors always rest between two elements in the list, and index in a logically circular way. To accommodate this, there is a "ghost" non-element that yields None between the head and tail of the list.
HEY WAIT. This is the opposite of what the RFC says??? But wait all the docs on the methods still refer to "current" elements... wait hold on, where have I seen this ghost stuff before. Oh wait, didn't I do that in my old linked-list fork where I prototyped?
Cursors always rest between two elements in the list, and index in a logically circular way. To accomadate this, there is a "ghost" non-element that yields None between the head and tail of the List.
Hold up what the fuck. This isn't a gag, I am actually trying to Read The Docs right now. Did std actually RFC a different design from the one I proposed in 2015, but then copy-paste the docs from my prototype??? Is std meta-shitposting me for writing a book about how much I hate LinkedList????? Like yeah I built that prototype to demonstrate the concept so that people would let me add it to std and make LinkedList not useless but, qu'est-ce que le fuck??????????????
Ok you know what, clearly std is blessing my design as the objectively superior one, so we're going to do my design. Also that's nice because this entire chapter is me actually literally rewriting that library from scratch, so not changing the API sounds Good To Me!
Here's the full top-level docs I wrote:
A Cursor is like an iterator, except that it can freely seek back-and-forth, and can safely mutate the list during iteration. This is because the lifetime of its yielded references are tied to its own lifetime, instead of just the underlying list. This means cursors cannot yield multiple elements at once.
Cursors always rest between two elements in the list, and index in a logically circular way. To accomadate this, there is a "ghost" non-element that yields None between the head and tail of the List.
When created, cursors start between the ghost and the front of the list. That is, next will yield the front of the list, and prev will yield None. Calling prev again will yield the tail.
Cute, even though we concluded that the whole "sentinel-node" thing was more trouble than it's worth, we're still going to end up with semantics that "pretend" there's a sentinel node so that the cursor can wrap around to the other side of the list.
Skims over my old APIs some more
fn splice(&mut self, other: &mut LinkedList<T>)
Inserts the entire list's contents right after the cursor.
Oh yeah, this is coming back to me. I wrote this when I was really mad about combinatoric explosion, and was trying to come up with a way for there to only be one copy of each operation. Unfortunately this is... semantically problematic. See, when the user wants to splice one list into another, they might want the cursor to end up before the splice or after it. The inserted list can be arbitrarily large, so it's a genuine issue for us to only allow for one and expect the user to walk over the entire inserted list!
We're gonna have to rework this design from the ground up after all. What does our Cursor type need? Well it needs to:
- point "between" two elements
- as a nice little feature, keep track of what "index" is next
- update the list itself to modify front/back/len.
How do you point between two elements? Well, you don't. You just point at the "next" element. So, yeah even though we're exposing "cursor goes in-between" semantics, we're really implementing it as "cursor is on", and just pretending everything happens before or after that point.
But there's a reason! The splice use-case wants to let the user choose whether they end up before or after the list, but this is... horribly complicated to express with the std API! They have splice_after and splice_before, but neither changes the cursor's position, so really you'd need splice_after_before and splice_after_after...
Wait no I'm being silly. In the std API you can just choose the node you want to end up on, and then use splice_after/before as appropriate.
squints
Wait is the std API actually good.
skims through the code
Ok the std API is actually good.
Alright screw it, we're going to implement the RFC. Or at least the interesting parts of it.
I have my quibbles with some of the terminology std uses, but cursors are always going to be a bit brain-melty: iter().next_back() gets you back(), so that's good, but then each subsequent next_back() is actually bringing you closer to the front and indeed, every pointer we follow is a "front" pointer! If I think about this seeming-paradox too much it hurts my brain, so, I can certainly respect going for different terminology to avoid this.
The std API talks about operations before "before" (towards the front) and "after" (towards the back), and instead of next and next_back, it... calls things move_next and move_prev. HRM. Ok so they're getting into a bit of the iterator terminology, but at least next doesn't evoke front/back, and helps you orient how things behave compared to the iterators.
We can work with this.
Implementing Cursors
Ok so we're only going to bother with std's CursorMut because the immutable version isn't actually interesting. Just like my original design, it has a "ghost" element that contains None to indicate the start/end of the list, and you can "walk over it" to wrap around to the other side of the list. To implement it, we're going to need:
- A pointer to the current node
- A pointer to the list
- The current index
Wait what's the index when we point at the "ghost"?
furrows brow ... checks std ... dislikes std's answer
Ok so quite reasonably index on a Cursor returns an Option<usize>. The std implementation does a bunch of junk to avoid storing it as an Option but... we're a linked list, it's fine. Also std has the cursor_front/cursor_back stuff which starts the cursor on the front/back elements, which feels intuitive, but then has to do something weird when the list is empty.
You can implement that stuff if you want, but I'm going to cut down on all the repetitive gunk and corner cases and just make a bare cursor_mut method that starts at the ghost, and people can use move_next/move_prev to get the one they want (and then you can wrap that up as cursor_front if you really want).
Let's get cracking:
pub struct CursorMut<'a, T> {
cur: Link<T>,
list: &'a mut LinkedList<T>,
index: Option<usize>,
}
Pretty straight-forward, one field for each item of our bulleted list! Now the cursor_mut method:
impl<T> LinkedList<T> {
pub fn cursor_mut(&mut self) -> CursorMut<T> {
CursorMut {
list: self,
cur: None,
index: None,
}
}
}
Since we're starting at the ghost, we can just start with everything as None, nice and simple! Next, movement:
impl<'a, T> CursorMut<'a, T> {
pub fn index(&self) -> Option<usize> {
self.index
}
pub fn move_next(&mut self) {
if let Some(cur) = self.cur {
unsafe {
// We're on a real element, go to its next (back)
self.cur = (*cur.as_ptr()).back;
if self.cur.is_some() {
*self.index.as_mut().unwrap() += 1;
} else {
// We just walked to the ghost, no more index
self.index = None;
}
}
} else if !self.list.is_empty() {
// We're at the ghost, and there is a real front, so move to it!
self.cur = self.list.front;
self.index = Some(0)
} else {
// We're at the ghost, but that's the only element... do nothing.
}
}
}
So there's 4 interesting cases:
- The normal case
- The normal case, but we reach the ghost
- The ghost case, where we go to the front of the list
- The ghost case, but the list is empty, so do nothing
move_prev is the exact same logic, but with front/back inverted and the indexing changes inverted:
pub fn move_prev(&mut self) {
if let Some(cur) = self.cur {
unsafe {
// We're on a real element, go to its previous (front)
self.cur = (*cur.as_ptr()).front;
if self.cur.is_some() {
*self.index.as_mut().unwrap() -= 1;
} else {
// We just walked to the ghost, no more index
self.index = None;
}
}
} else if !self.list.is_empty() {
// We're at the ghost, and there is a real back, so move to it!
self.cur = self.list.back;
self.index = Some(self.list.len - 1)
} else {
// We're at the ghost, but that's the only element... do nothing.
}
}
Next let's add some methods to look at the elements around the cursor: current, peek_next, and peek_prev. A Very Important Note: these methods must borrow our cursor by &mut self, and the results must be tied to that borrow. We cannot let the user get multiple copies of a mutable reference, and we cannot let them use any of our insert/remove/split/splice APIs while holding onto such a reference!
Thankfully, this is the default assumption rust makes when you use lifetime elision, so, we will just do the right thing by default!
pub fn current(&mut self) -> Option<&mut T> {
unsafe {
self.cur.map(|node| &mut (*node.as_ptr()).elem)
}
}
pub fn peek_next(&mut self) -> Option<&mut T> {
unsafe {
self.cur
.and_then(|node| (*node.as_ptr()).back)
.map(|node| &mut (*node.as_ptr()).elem)
}
}
pub fn peek_prev(&mut self) -> Option<&mut T> {
unsafe {
self.cur
.and_then(|node| (*node.as_ptr()).front)
.map(|node| &mut (*node.as_ptr()).elem)
}
}
Head empty, Option methods and (omitted) compiler errors do all thinking now. I was skeptical about the Option<NonNull> stuff, but, god damn it really just lets me autopilot this code. I've spent way too much time writing array-based collections where you never get to use Option, wow this is nice! ((*node.as_ptr()) is still miserable but, that's just Rust's raw pointers for you...)
Next we have a choice: we can either jump right to split and splice, the entire point of these APIs, or we can take a baby-step with single element insert/remove. I have a feeling we're just going to want to implement insert/remove in terms of split and splice so... let's just do those first and see where the cards fall (genuinely have no idea as I type this).
Split
First up, split_before and split_after, which return everything before/after the current element as a LinkedList (stopping at the ghost element, unless you're at the ghost, in which case we just return the whole List and the cursor now points to an empty list):
squints ok this one is actually some non-trivial logic so we're going to have to talk it out one step at a time.
I see 4 potentially interesting cases for split_before:
- The normal case
- The normal case, but prev is the ghost
- The ghost case, where we return the whole list and become empty
- The ghost case, but the list is empty, so do nothing and return the empty list
Let's start with the corner cases. The third case I believe is just
#![allow(unused)] fn main() { mem::replace(self.list, LinkedList::new()) }
Right? We become empty, we return the whole list, and our fields were already None, so nothing to update. Nice. Oh hey, this also Does The Right Thing on the fourth case too!
So now the normal cases... ok I'm going to need some ASCII diagrams for this. In the most general case, we have something like this:
list.front -> A <-> B <-> C <-> D <- list.back
^
cur
And we want to produce this:
list.front -> C <-> D <- list.back
^
cur
return.front -> A <-> B <- return.back
So we need to break the link between cur and prev, and... god so much needs to change. Ok I just need to break this up into steps so I can convince myself it makes sense. This will be a bit over-verbose but I can at least make sense of it:
pub fn split_before(&mut self) -> LinkedList<T> {
if let Some(cur) = self.cur {
// We are pointing at a real element, so the list is non-empty.
unsafe {
// Current state
let old_len = self.list.len;
let old_idx = self.index.unwrap();
let prev = (*cur.as_ptr()).front;
// What self will become
let new_len = old_len - old_idx;
let new_front = self.cur;
let new_back = self.list.back;
let new_idx = Some(0);
// What the output will become
let output_len = old_len - new_len;
let output_front = self.list.front;
let output_back = prev;
// Break the links between cur and prev
if let Some(prev) = prev {
(*cur.as_ptr()).front = None;
(*prev.as_ptr()).back = None;
}
// Produce the result:
self.list.len = new_len;
self.list.front = new_front;
self.list.back = new_back;
self.index = new_idx;
LinkedList {
front: output_front,
back: output_back,
len: output_len,
_boo: PhantomData,
}
}
} else {
// We're at the ghost, just replace our list with an empty one.
// No other state needs to be changed.
std::mem::replace(self.list, LinkedList::new())
}
}
Note that this if-let is handling the "normal case, but prev is the ghost" situation:
if let Some(prev) = prev {
(*cur.as_ptr()).front = None;
(*prev.as_ptr()).back = None;
}
If you want to, you can squash that all together and apply optimizations like:
- fold the two accesses to
(*cur.as_ptr()).frontas just(*cur.as_ptr()).front.take() - note that new_back is a noop, and just remove both
As far as I can tell, everything else just incidentally Does The Right Thing otherwise. We'll see when we write tests! (copy-paste to make split_after)
I am done Making Mistakes and I am just going to try to write the most foolproof code I can. This is how I actually write collections: just break things down into trivial steps and cases until it can fit in my head and seems foolproof. Then write a ton of tests until I'm convinced I didn't manage to mess it up still.
Because most of the collections work I've done is extremely unsafe I don't generally get to rely on the compiler catching mistakes, and miri didn't exist back in the day! So I just need to squint at a problem until my head hurts and try my hardest to Never Ever Ever Make A Mistake.
Don't write Unsafe Rust Code! Safe Rust is so much better!!!!
Splice
Just one more boss to fight, splice_before and splice_after, which I expect to be the corner-casiest one of them all. The two functions take in a LinkedList and grafts its contents into outrs. Our list could be empty, their list could be empty, we've got ghosts to deal with... sigh let's just take it one step at a time with splice_before.
- If their list is empty, we don't need to do anything.
- If our list is empty, then our list just becomes their list.
- If we're pointing at the ghost, then this appends to the back (change list.back)
- If we're pointing at the first element (0), this this appends to the front (change list.front)
- In the general case, we do a whole lot of pointer fuckery.
The general case is this:
input.front -> 1 <-> 2 <- input.back
list.front -> A <-> B <-> C <- list.back
^
cur
Becoming this:
list.front -> A <-> 1 <-> 2 <-> B <-> C <- list.back
Ok? Ok. Let's write that out... TAKES A HUGE BREATH AND PLUNGES IN:
pub fn splice_before(&mut self, mut input: LinkedList<T>) {
unsafe {
if input.is_empty() {
// Input is empty, do nothing.
} else if let Some(cur) = self.cur {
if let Some(0) = self.index {
// We're appending to the front, see append to back
(*cur.as_ptr()).front = input.back.take();
(*input.back.unwrap().as_ptr()).back = Some(cur);
self.list.front = input.front.take();
// Index moves forward by input length
*self.index.as_mut().unwrap() += input.len;
self.list.len += input.len;
input.len = 0;
} else {
// General Case, no boundaries, just internal fixups
let prev = (*cur.as_ptr()).front.unwrap();
let in_front = input.front.take().unwrap();
let in_back = input.back.take().unwrap();
(*prev.as_ptr()).back = Some(in_front);
(*in_front.as_ptr()).front = Some(prev);
(*cur.as_ptr()).front = Some(in_back);
(*in_back.as_ptr()).back = Some(cur);
// Index moves forward by input length
*self.index.as_mut().unwrap() += input.len;
self.list.len += input.len;
input.len = 0;
}
} else if let Some(back) = self.list.back {
// We're on the ghost but non-empty, append to the back
// We can either `take` the input's pointers or `mem::forget`
// it. Using take is more responsible in case we do custom
// allocators or something that also needs to be cleaned up!
(*back.as_ptr()).back = input.front.take();
(*input.front.unwrap().as_ptr()).front = Some(back);
self.list.back = input.back.take();
self.list.len += input.len;
// Not necessary but Polite To Do
input.len = 0;
} else {
// We're empty, become the input, remain on the ghost
*self.list = input;
}
}
}
Ok this one is genuinely horrendous, and really is feeling that Option<NonNull> pain now. But there's a lot of cleanups we can do. For one, we can pull this code out to the very end, because we always want to do it. I don't love (although sometimes it's a noop, and setting input.len is more a matter of paranoia about future extensions to the code):
self.list.len += input.len;
input.len = 0;
Use of moved value:
input
Ah, right, in the "we're empty" case we're moving the list. Let's replace that with a swap:
// We're empty, become the input, remain on the ghost
std::mem::swap(self.list, &mut input);
In this case the writes will be pointless, but, they still work (we could probably also early-return in this branch to appease the compiler).
This unwrap is just a consequence of me thinking about the cases backwards, and can be fixed by making the if-let ask the right question:
if let Some(0) = self.index {
} else {
let prev = (*cur.as_ptr()).front.unwrap();
}
Adjusting the index is duplicated inside the branches, so can also be hoisted out:
#![allow(unused)] fn main() { *self.index.as_mut().unwrap() += input.len; }
Ok, putting that all together we get this:
#![allow(unused)] fn main() { if input.is_empty() { // Input is empty, do nothing. } else if let Some(cur) = self.cur { // Both lists are non-empty if let Some(prev) = (*cur.as_ptr()).front { // General Case, no boundaries, just internal fixups let in_front = input.front.take().unwrap(); let in_back = input.back.take().unwrap(); (*prev.as_ptr()).back = Some(in_front); (*in_front.as_ptr()).front = Some(prev); (*cur.as_ptr()).front = Some(in_back); (*in_back.as_ptr()).back = Some(cur); } else { // We're appending to the front, see append to back below (*cur.as_ptr()).front = input.back.take(); (*input.back.unwrap().as_ptr()).back = Some(cur); self.list.front = input.front.take(); } // Index moves forward by input length *self.index.as_mut().unwrap() += input.len; } else if let Some(back) = self.list.back { // We're on the ghost but non-empty, append to the back // We can either `take` the input's pointers or `mem::forget` // it. Using take is more responsible in case we do custom // allocators or something that also needs to be cleaned up! (*back.as_ptr()).back = input.front.take(); (*input.front.unwrap().as_ptr()).front = Some(back); self.list.back = input.back.take(); } else { // We're empty, become the input, remain on the ghost std::mem::swap(self.list, &mut input); } self.list.len += input.len; // Not necessary but Polite To Do input.len = 0; // Input dropped here }
Alright this still sucks, but mostly because of -- nope ok just spotted a bug:
#![allow(unused)] fn main() { (*back.as_ptr()).back = input.front.take(); (*input.front.unwrap().as_ptr()).front = Some(back); }
We take input.front and then unwrap it on the next line! sigh and we do the same thing in the equivalent mirror case. We would have caught this instantly in tests, but, we're trying to be Perfect now, and I'm just kinda doing this live, and this is the exact moment where I saw it. This is what I get for not being my usual tedious self and doing things in phases. More explicit!
#![allow(unused)] fn main() { // We can either `take` the input's pointers or `mem::forget` // it. Using `take` is more responsible in case we ever do custom // allocators or something that also needs to be cleaned up! if input.is_empty() { // Input is empty, do nothing. } else if let Some(cur) = self.cur { // Both lists are non-empty let in_front = input.front.take().unwrap(); let in_back = input.back.take().unwrap(); if let Some(prev) = (*cur.as_ptr()).front { // General Case, no boundaries, just internal fixups (*prev.as_ptr()).back = Some(in_front); (*in_front.as_ptr()).front = Some(prev); (*cur.as_ptr()).front = Some(in_back); (*in_back.as_ptr()).back = Some(cur); } else { // No prev, we're appending to the front (*cur.as_ptr()).front = Some(in_back); (*in_back.as_ptr()).back = Some(cur); self.list.front = Some(in_front); } // Index moves forward by input length *self.index.as_mut().unwrap() += input.len; } else if let Some(back) = self.list.back { // We're on the ghost but non-empty, append to the back let in_front = input.front.take().unwrap(); let in_back = input.back.take().unwrap(); (*back.as_ptr()).back = Some(in_front); (*in_front.as_ptr()).front = Some(back); self.list.back = Some(in_back); } else { // We're empty, become the input, remain on the ghost std::mem::swap(self.list, &mut input); } self.list.len += input.len; // Not necessary but Polite To Do input.len = 0; // Input dropped here }
Alright now this, this I can tolerate. The only complaints I have are that we don't dedupe in_front/in_back (probably we could rejig our conditions but eh whatever). Really this is basically what you would write in C but with Option<NonNull> gunk making it tedious. I can live with that. Well no we should just make raw pointers better for this stuff. But, out of scope for this book.
Anyway, I am absolutely exhausted after that, so, insert and remove and all the other APIs can be left as an excercise to the reader.
Here's the final code for our Cursor with my attempt at copy-pasting the combinatorics. Did I get it right? I'll only find out when I write the next chapter and test this monstrosity!
pub struct CursorMut<'a, T> {
list: &'a mut LinkedList<T>,
cur: Link<T>,
index: Option<usize>,
}
impl<T> LinkedList<T> {
pub fn cursor_mut(&mut self) -> CursorMut<T> {
CursorMut {
list: self,
cur: None,
index: None,
}
}
}
impl<'a, T> CursorMut<'a, T> {
pub fn index(&self) -> Option<usize> {
self.index
}
pub fn move_next(&mut self) {
if let Some(cur) = self.cur {
unsafe {
// We're on a real element, go to its next (back)
self.cur = (*cur.as_ptr()).back;
if self.cur.is_some() {
*self.index.as_mut().unwrap() += 1;
} else {
// We just walked to the ghost, no more index
self.index = None;
}
}
} else if !self.list.is_empty() {
// We're at the ghost, and there is a real front, so move to it!
self.cur = self.list.front;
self.index = Some(0)
} else {
// We're at the ghost, but that's the only element... do nothing.
}
}
pub fn move_prev(&mut self) {
if let Some(cur) = self.cur {
unsafe {
// We're on a real element, go to its previous (front)
self.cur = (*cur.as_ptr()).front;
if self.cur.is_some() {
*self.index.as_mut().unwrap() -= 1;
} else {
// We just walked to the ghost, no more index
self.index = None;
}
}
} else if !self.list.is_empty() {
// We're at the ghost, and there is a real back, so move to it!
self.cur = self.list.back;
self.index = Some(self.list.len - 1)
} else {
// We're at the ghost, but that's the only element... do nothing.
}
}
pub fn current(&mut self) -> Option<&mut T> {
unsafe {
self.cur.map(|node| &mut (*node.as_ptr()).elem)
}
}
pub fn peek_next(&mut self) -> Option<&mut T> {
unsafe {
self.cur
.and_then(|node| (*node.as_ptr()).back)
.map(|node| &mut (*node.as_ptr()).elem)
}
}
pub fn peek_prev(&mut self) -> Option<&mut T> {
unsafe {
self.cur
.and_then(|node| (*node.as_ptr()).front)
.map(|node| &mut (*node.as_ptr()).elem)
}
}
pub fn split_before(&mut self) -> LinkedList<T> {
// We have this:
//
// list.front -> A <-> B <-> C <-> D <- list.back
// ^
// cur
//
//
// And we want to produce this:
//
// list.front -> C <-> D <- list.back
// ^
// cur
//
//
// return.front -> A <-> B <- return.back
//
if let Some(cur) = self.cur {
// We are pointing at a real element, so the list is non-empty.
unsafe {
// Current state
let old_len = self.list.len;
let old_idx = self.index.unwrap();
let prev = (*cur.as_ptr()).front;
// What self will become
let new_len = old_len - old_idx;
let new_front = self.cur;
let new_back = self.list.back;
let new_idx = Some(0);
// What the output will become
let output_len = old_len - new_len;
let output_front = self.list.front;
let output_back = prev;
// Break the links between cur and prev
if let Some(prev) = prev {
(*cur.as_ptr()).front = None;
(*prev.as_ptr()).back = None;
}
// Produce the result:
self.list.len = new_len;
self.list.front = new_front;
self.list.back = new_back;
self.index = new_idx;
LinkedList {
front: output_front,
back: output_back,
len: output_len,
_boo: PhantomData,
}
}
} else {
// We're at the ghost, just replace our list with an empty one.
// No other state needs to be changed.
std::mem::replace(self.list, LinkedList::new())
}
}
pub fn split_after(&mut self) -> LinkedList<T> {
// We have this:
//
// list.front -> A <-> B <-> C <-> D <- list.back
// ^
// cur
//
//
// And we want to produce this:
//
// list.front -> A <-> B <- list.back
// ^
// cur
//
//
// return.front -> C <-> D <- return.back
//
if let Some(cur) = self.cur {
// We are pointing at a real element, so the list is non-empty.
unsafe {
// Current state
let old_len = self.list.len;
let old_idx = self.index.unwrap();
let next = (*cur.as_ptr()).back;
// What self will become
let new_len = old_idx + 1;
let new_back = self.cur;
let new_front = self.list.front;
let new_idx = Some(old_idx);
// What the output will become
let output_len = old_len - new_len;
let output_front = next;
let output_back = self.list.back;
// Break the links between cur and next
if let Some(next) = next {
(*cur.as_ptr()).back = None;
(*next.as_ptr()).front = None;
}
// Produce the result:
self.list.len = new_len;
self.list.front = new_front;
self.list.back = new_back;
self.index = new_idx;
LinkedList {
front: output_front,
back: output_back,
len: output_len,
_boo: PhantomData,
}
}
} else {
// We're at the ghost, just replace our list with an empty one.
// No other state needs to be changed.
std::mem::replace(self.list, LinkedList::new())
}
}
pub fn splice_before(&mut self, mut input: LinkedList<T>) {
// We have this:
//
// input.front -> 1 <-> 2 <- input.back
//
// list.front -> A <-> B <-> C <- list.back
// ^
// cur
//
//
// Becoming this:
//
// list.front -> A <-> 1 <-> 2 <-> B <-> C <- list.back
// ^
// cur
//
unsafe {
// We can either `take` the input's pointers or `mem::forget`
// it. Using `take` is more responsible in case we ever do custom
// allocators or something that also needs to be cleaned up!
if input.is_empty() {
// Input is empty, do nothing.
} else if let Some(cur) = self.cur {
// Both lists are non-empty
let in_front = input.front.take().unwrap();
let in_back = input.back.take().unwrap();
if let Some(prev) = (*cur.as_ptr()).front {
// General Case, no boundaries, just internal fixups
(*prev.as_ptr()).back = Some(in_front);
(*in_front.as_ptr()).front = Some(prev);
(*cur.as_ptr()).front = Some(in_back);
(*in_back.as_ptr()).back = Some(cur);
} else {
// No prev, we're appending to the front
(*cur.as_ptr()).front = Some(in_back);
(*in_back.as_ptr()).back = Some(cur);
self.list.front = Some(in_front);
}
// Index moves forward by input length
*self.index.as_mut().unwrap() += input.len;
} else if let Some(back) = self.list.back {
// We're on the ghost but non-empty, append to the back
let in_front = input.front.take().unwrap();
let in_back = input.back.take().unwrap();
(*back.as_ptr()).back = Some(in_front);
(*in_front.as_ptr()).front = Some(back);
self.list.back = Some(in_back);
} else {
// We're empty, become the input, remain on the ghost
std::mem::swap(self.list, &mut input);
}
self.list.len += input.len;
// Not necessary but Polite To Do
input.len = 0;
// Input dropped here
}
}
pub fn splice_after(&mut self, mut input: LinkedList<T>) {
// We have this:
//
// input.front -> 1 <-> 2 <- input.back
//
// list.front -> A <-> B <-> C <- list.back
// ^
// cur
//
//
// Becoming this:
//
// list.front -> A <-> B <-> 1 <-> 2 <-> C <- list.back
// ^
// cur
//
unsafe {
// We can either `take` the input's pointers or `mem::forget`
// it. Using `take` is more responsible in case we ever do custom
// allocators or something that also needs to be cleaned up!
if input.is_empty() {
// Input is empty, do nothing.
} else if let Some(cur) = self.cur {
// Both lists are non-empty
let in_front = input.front.take().unwrap();
let in_back = input.back.take().unwrap();
if let Some(next) = (*cur.as_ptr()).back {
// General Case, no boundaries, just internal fixups
(*next.as_ptr()).front = Some(in_back);
(*in_back.as_ptr()).back = Some(next);
(*cur.as_ptr()).back = Some(in_front);
(*in_front.as_ptr()).front = Some(cur);
} else {
// No next, we're appending to the back
(*cur.as_ptr()).back = Some(in_front);
(*in_front.as_ptr()).front = Some(cur);
self.list.back = Some(in_back);
}
// Index doesn't change
} else if let Some(front) = self.list.front {
// We're on the ghost but non-empty, append to the front
let in_front = input.front.take().unwrap();
let in_back = input.back.take().unwrap();
(*front.as_ptr()).front = Some(in_back);
(*in_back.as_ptr()).back = Some(front);
self.list.front = Some(in_front);
} else {
// We're empty, become the input, remain on the ghost
std::mem::swap(self.list, &mut input);
}
self.list.len += input.len;
// Not necessary but Polite To Do
input.len = 0;
// Input dropped here
}
}
}
Testing Cursors
Time to find out how many horribly embarassing mistakes I made in the previous section!
Oh god we made our API unlike both std and the old impl. Alright well I'm just gonna hastily cobble together something from both of them. Yeah let's "borrow" these tests from std:
#[test]
fn test_cursor_move_peek() {
let mut m: LinkedList<u32> = LinkedList::new();
m.extend([1, 2, 3, 4, 5, 6]);
let mut cursor = m.cursor_mut();
cursor.move_next();
assert_eq!(cursor.current(), Some(&mut 1));
assert_eq!(cursor.peek_next(), Some(&mut 2));
assert_eq!(cursor.peek_prev(), None);
assert_eq!(cursor.index(), Some(0));
cursor.move_prev();
assert_eq!(cursor.current(), None);
assert_eq!(cursor.peek_next(), Some(&mut 1));
assert_eq!(cursor.peek_prev(), Some(&mut 6));
assert_eq!(cursor.index(), None);
cursor.move_next();
cursor.move_next();
assert_eq!(cursor.current(), Some(&mut 2));
assert_eq!(cursor.peek_next(), Some(&mut 3));
assert_eq!(cursor.peek_prev(), Some(&mut 1));
assert_eq!(cursor.index(), Some(1));
let mut cursor = m.cursor_mut();
cursor.move_prev();
assert_eq!(cursor.current(), Some(&mut 6));
assert_eq!(cursor.peek_next(), None);
assert_eq!(cursor.peek_prev(), Some(&mut 5));
assert_eq!(cursor.index(), Some(5));
cursor.move_next();
assert_eq!(cursor.current(), None);
assert_eq!(cursor.peek_next(), Some(&mut 1));
assert_eq!(cursor.peek_prev(), Some(&mut 6));
assert_eq!(cursor.index(), None);
cursor.move_prev();
cursor.move_prev();
assert_eq!(cursor.current(), Some(&mut 5));
assert_eq!(cursor.peek_next(), Some(&mut 6));
assert_eq!(cursor.peek_prev(), Some(&mut 4));
assert_eq!(cursor.index(), Some(4));
}
#[test]
fn test_cursor_mut_insert() {
let mut m: LinkedList<u32> = LinkedList::new();
m.extend([1, 2, 3, 4, 5, 6]);
let mut cursor = m.cursor_mut();
cursor.move_next();
cursor.splice_before(Some(7).into_iter().collect());
cursor.splice_after(Some(8).into_iter().collect());
// check_links(&m);
assert_eq!(m.iter().cloned().collect::<Vec<_>>(), &[7, 1, 8, 2, 3, 4, 5, 6]);
let mut cursor = m.cursor_mut();
cursor.move_next();
cursor.move_prev();
cursor.splice_before(Some(9).into_iter().collect());
cursor.splice_after(Some(10).into_iter().collect());
check_links(&m);
assert_eq!(m.iter().cloned().collect::<Vec<_>>(), &[10, 7, 1, 8, 2, 3, 4, 5, 6, 9]);
/* remove_current not impl'd
let mut cursor = m.cursor_mut();
cursor.move_next();
cursor.move_prev();
assert_eq!(cursor.remove_current(), None);
cursor.move_next();
cursor.move_next();
assert_eq!(cursor.remove_current(), Some(7));
cursor.move_prev();
cursor.move_prev();
cursor.move_prev();
assert_eq!(cursor.remove_current(), Some(9));
cursor.move_next();
assert_eq!(cursor.remove_current(), Some(10));
check_links(&m);
assert_eq!(m.iter().cloned().collect::<Vec<_>>(), &[1, 8, 2, 3, 4, 5, 6]);
*/
let mut cursor = m.cursor_mut();
cursor.move_next();
let mut p: LinkedList<u32> = LinkedList::new();
p.extend([100, 101, 102, 103]);
let mut q: LinkedList<u32> = LinkedList::new();
q.extend([200, 201, 202, 203]);
cursor.splice_after(p);
cursor.splice_before(q);
check_links(&m);
assert_eq!(
m.iter().cloned().collect::<Vec<_>>(),
&[200, 201, 202, 203, 1, 100, 101, 102, 103, 8, 2, 3, 4, 5, 6]
);
let mut cursor = m.cursor_mut();
cursor.move_next();
cursor.move_prev();
let tmp = cursor.split_before();
assert_eq!(m.into_iter().collect::<Vec<_>>(), &[]);
m = tmp;
let mut cursor = m.cursor_mut();
cursor.move_next();
cursor.move_next();
cursor.move_next();
cursor.move_next();
cursor.move_next();
cursor.move_next();
cursor.move_next();
let tmp = cursor.split_after();
assert_eq!(tmp.into_iter().collect::<Vec<_>>(), &[102, 103, 8, 2, 3, 4, 5, 6]);
check_links(&m);
assert_eq!(m.iter().cloned().collect::<Vec<_>>(), &[200, 201, 202, 203, 1, 100, 101]);
}
fn check_links<T>(_list: &LinkedList<T>) {
// would be good to do this!
}
Moment of truth!
cargo test
Compiling linked-list v0.0.3
Finished test [unoptimized + debuginfo] target(s) in 1.03s
Running unittests src\lib.rs
running 14 tests
test test::test_basic_front ... ok
test test::test_basic ... ok
test test::test_debug ... ok
test test::test_iterator_mut_double_end ... ok
test test::test_ord ... ok
test test::test_cursor_move_peek ... FAILED
test test::test_cursor_mut_insert ... FAILED
test test::test_iterator ... ok
test test::test_mut_iter ... ok
test test::test_eq ... ok
test test::test_rev_iter ... ok
test test::test_iterator_double_end ... ok
test test::test_hashmap ... ok
test test::test_ord_nan ... ok
failures:
---- test::test_cursor_move_peek stdout ----
thread 'test::test_cursor_move_peek' panicked at 'assertion failed: `(left == right)`
left: `None`,
right: `Some(1)`', src\lib.rs:1079:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- test::test_cursor_mut_insert stdout ----
thread 'test::test_cursor_mut_insert' panicked at 'assertion failed: `(left == right)`
left: `[200, 201, 202, 203, 10, 100, 101, 102, 103, 7, 1, 8, 2, 3, 4, 5, 6, 9]`,
right: `[200, 201, 202, 203, 1, 100, 101, 102, 103, 8, 2, 3, 4, 5, 6]`', src\lib.rs:1153:9
failures:
test::test_cursor_move_peek
test::test_cursor_mut_insert
test result: FAILED. 12 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
I'll admit, I had some hubris here and was hoping I nailed it. This is why we write tests (but maybe I just did a bad job of porting the tests..?).
What's the first failure?
let mut m: LinkedList<u32> = LinkedList::new();
m.extend([1, 2, 3, 4, 5, 6]);
let mut cursor = m.cursor_mut();
cursor.move_next();
assert_eq!(cursor.current(), Some(&mut 1));
assert_eq!(cursor.peek_next(), Some(&mut 2));
assert_eq!(cursor.peek_prev(), None);
assert_eq!(cursor.index(), Some(0));
cursor.move_prev();
assert_eq!(cursor.current(), None);
assert_eq!(cursor.peek_next(), Some(&mut 1)); // DIES HERE
Geez I really messed up some basic functionality. Wait,
Head empty, Option methods and (omitted) compiler errors do all thinking now.
Well I am nothing if not honest.
pub fn peek_next(&mut self) -> Option<&mut T> {
unsafe {
self.cur
.and_then(|node| (*node.as_ptr()).back)
.map(|node| &mut (*node.as_ptr()).elem)
}
}
...Yeah this is just wrong. If self.cur is None, we aren't just supposed to give up, we need to check self.list.front too, because we're on the ghost! So we just need to add an or_else to the chain:
pub fn peek_next(&mut self) -> Option<&mut T> {
unsafe {
self.cur
.and_then(|node| (*node.as_ptr()).back)
.or_else(|| self.list.front)
.map(|node| &mut (*node.as_ptr()).elem)
}
}
pub fn peek_prev(&mut self) -> Option<&mut T> {
unsafe {
self.cur
.and_then(|node| (*node.as_ptr()).front)
.or_else(|| self.list.back)
.map(|node| &mut (*node.as_ptr()).elem)
}
}
Did that fix it?
---- test::test_cursor_move_peek stdout ----
thread 'test::test_cursor_move_peek' panicked at 'assertion failed: `(left == right)`
left: `Some(6)`,
right: `None`', src\lib.rs:1078:9
Wait now it's wrong further back. Ok I need to stop head-emptying peek because apparently it's a lot harder than I was willing to give it credit for. Just trying to blindly chain these cases is a disaster, let's have a proper if for the cases of ghost vs not:
pub fn peek_next(&mut self) -> Option<&mut T> {
unsafe {
let next = if let Some(cur) = self.cur {
// Normal case, try to follow the cur node's back pointer
(*cur.as_ptr()).back
} else {
// Ghost case, try to use the list's front pointer
self.list.front
};
// Yield the element if the next node exists
next.map(|node| &mut (*node.as_ptr()).elem)
}
}
pub fn peek_prev(&mut self) -> Option<&mut T> {
unsafe {
let prev = if let Some(cur) = self.cur {
// Normal case, try to follow the cur node's front pointer
(*cur.as_ptr()).front
} else {
// Ghost case, try to use the list's back pointer
self.list.back
};
// Yield the element if the prev node exists
prev.map(|node| &mut (*node.as_ptr()).elem)
}
}
Feelin' confident about this one!
failures:
---- test::test_cursor_mut_insert stdout ----
thread 'test::test_cursor_mut_insert' panicked at 'assertion failed: `(left == right)`
left: `[200, 201, 202, 203, 10, 100, 101, 102, 103, 7, 1, 8, 2, 3, 4, 5, 6, 9]`,
right: `[200, 201, 202, 203, 1, 100, 101, 102, 103, 8, 2, 3, 4, 5, 6]`', src\lib.rs:1168:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
test::test_cursor_mut_insert
test result: FAILED. 13 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Yesss. Ok one more failure to go... oh.
Did you notice the part where I commented out some code for testing remove_current? Yeah I wasn't paying attention to the fact that this test is stateful. Let's just create a new list with the state the remove_current part would have left us in:
let mut m: LinkedList<u32> = LinkedList::new();
m.extend([1, 8, 2, 3, 4, 5, 6]);
cargo test
Compiling linked-list v0.0.3
Finished test [unoptimized + debuginfo] target(s) in 0.70s
Running unittests src\lib.rs
running 14 tests
test test::test_basic_front ... ok
test test::test_basic ... ok
test test::test_cursor_move_peek ... ok
test test::test_eq ... ok
test test::test_cursor_mut_insert ... ok
test test::test_iterator ... ok
test test::test_iterator_double_end ... ok
test test::test_ord_nan ... ok
test test::test_mut_iter ... ok
test test::test_hashmap ... ok
test test::test_debug ... ok
test test::test_ord ... ok
test test::test_iterator_mut_double_end ... ok
test test::test_rev_iter ... ok
test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Doc-tests linked-list
running 1 test
test src\lib.rs - assert_properties::iter_mut_invariant (line 803) - compile fail ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s
Heyyyy look at thaaat... ok now I'm getting paranoid. Let's properly fill in check_links and test it under miri:
fn check_links<T: Eq + std::fmt::Debug>(list: &LinkedList<T>) {
let from_front: Vec<_> = list.iter().collect();
let from_back: Vec<_> = list.iter().rev().collect();
let re_reved: Vec<_> = from_back.into_iter().rev().collect();
assert_eq!(from_front, re_reved);
}
Is this the best way to do this? No. Is it fine? Yes.
$env:MIRIFLAGS="-Zmiri-tag-raw-pointers"
cargo miri test
Compiling linked-list v0.0.3
Finished test [unoptimized + debuginfo] target(s) in 0.25s
Running unittests src\lib.rs
running 14 tests
test test::test_basic ... ok
test test::test_basic_front ... ok
test test::test_cursor_move_peek ... ok
test test::test_cursor_mut_insert ... ok
test test::test_debug ... ok
test test::test_eq ... ok
test test::test_hashmap ... ok
test test::test_iterator ... ok
test test::test_iterator_double_end ... ok
test test::test_iterator_mut_double_end ... ok
test test::test_mut_iter ... ok
test test::test_ord ... ok
test test::test_ord_nan ... ok
test test::test_rev_iter ... ok
test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Doc-tests linked-list
running 1 test
test src\lib.rs - assert_properties::iter_mut_invariant (line 803) - compile fail ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s
DONE.
Done.
We did it. We made a god damn production-quality LinkedList, with basically all the same functionality as the one in std. Are we missing little convenience methods here and there? Absolutely. Will I add them into the final published version of the crate? Probably!
But, I am, So Very Tired.
So. We win.
Wait fuck. We're being production quality. Ok one last final boss: clippy.
cargo clippy
cargo clippy
Checking linked-list v0.0.3 (C:\Users\ninte\dev\contain\linked-list)
warning: redundant pattern matching, consider using `is_some()`
--> src\lib.rs:189:19
|
189 | while let Some(_) = self.pop_front() { }
| ----------^^^^^^^------------------- help: try this: `while self.pop_front().is_some()`
|
= note: `#[warn(clippy::redundant_pattern_matching)]` on by default
= note: this will change drop order of the result, as well as all temporaries
= note: add `#[allow(clippy::redundant_pattern_matching)]` if this is important
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_pattern_matching
warning: method `into_iter` can be confused for the standard trait method `std::iter::IntoIterator::into_iter`
--> src\lib.rs:210:5
|
210 | / pub fn into_iter(self) -> IntoIter<T> {
211 | | IntoIter {
212 | | list: self
213 | | }
214 | | }
| |_____^
|
= note: `#[warn(clippy::should_implement_trait)]` on by default
= help: consider implementing the trait `std::iter::IntoIterator` or choosing a less ambiguous method name
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#should_implement_trait
warning: redundant pattern matching, consider using `is_some()`
--> src\lib.rs:228:19
|
228 | while let Some(_) = self.pop_front() { }
| ----------^^^^^^^------------------- help: try this: `while self.pop_front().is_some()`
|
= note: this will change drop order of the result, as well as all temporaries
= note: add `#[allow(clippy::redundant_pattern_matching)]` if this is important
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_pattern_matching
warning: re-implementing `PartialEq::ne` is unnecessary
--> src\lib.rs:275:5
|
275 | / fn ne(&self, other: &Self) -> bool {
276 | | self.len() != other.len() || self.iter().ne(other)
277 | | }
| |_____^
|
= note: `#[warn(clippy::partialeq_ne_impl)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#partialeq_ne_impl
warning: `linked-list` (lib) generated 4 warnings
Finished dev [unoptimized + debuginfo] target(s) in 0.29s
Alright clippy, let's do this.
Complaint 1 (and 3): we use while let Some(_) = instead of while .is_some(). The loop is empty so this truly doesn't matter but ok fine, clippy, I'll do things your way.
Complaint 2: We have an actual inherent into_iter method. Wait, what checks std ok, point to clippy. IntoIterator is in the prelude (and basically a lang item) so, we don't need an inherent version too.
Complaint 4: we copied a weird cargocult from std. shrug fine I'll remove it.
cargo clippy
Finished dev [unoptimized + debuginfo] target(s) in 0.00s
Nice. Just one last thing to do before calling it production quality: fmt.
cargo fmt
...yeah it added some newlines and removed some trailing whitespace. Nothing interesting.
WE ARE NOW TRULY FINALLY DONE!!!!!!!!!!!!!!!!!!!!!
Final Code
I can't believe I actually just made you sit through me actually reimplementing std::collections::LinkedList from scratch, with all the fiddly little pedantry and mistakes I made along the way.
I did it, the book is done, I can finally rest.
Alright, here's all 1200 lines of our complete rewrite of in all of its glory. This should be the same text as this commit.
I'll put some polish and docs back on and publish 0.1.0 later.
#![allow(unused)] fn main() { use std::cmp::Ordering; use std::fmt::{self, Debug}; use std::hash::{Hash, Hasher}; use std::iter::FromIterator; use std::marker::PhantomData; use std::ptr::NonNull; pub struct LinkedList<T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<T>, } type Link<T> = Option<NonNull<Node<T>>>; struct Node<T> { front: Link<T>, back: Link<T>, elem: T, } pub struct Iter<'a, T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<&'a T>, } pub struct IterMut<'a, T> { front: Link<T>, back: Link<T>, len: usize, _boo: PhantomData<&'a mut T>, } pub struct IntoIter<T> { list: LinkedList<T>, } pub struct CursorMut<'a, T> { list: &'a mut LinkedList<T>, cur: Link<T>, index: Option<usize>, } impl<T> LinkedList<T> { pub fn new() -> Self { Self { front: None, back: None, len: 0, _boo: PhantomData, } } pub fn push_front(&mut self, elem: T) { // SAFETY: it's a linked-list, what do you want? unsafe { let new = NonNull::new_unchecked(Box::into_raw(Box::new(Node { front: None, back: None, elem, }))); if let Some(old) = self.front { // Put the new front before the old one (*old.as_ptr()).front = Some(new); (*new.as_ptr()).back = Some(old); } else { // If there's no front, then we're the empty list and need // to set the back too. self.back = Some(new); } // These things always happen! self.front = Some(new); self.len += 1; } } pub fn push_back(&mut self, elem: T) { // SAFETY: it's a linked-list, what do you want? unsafe { let new = NonNull::new_unchecked(Box::into_raw(Box::new(Node { back: None, front: None, elem, }))); if let Some(old) = self.back { // Put the new back before the old one (*old.as_ptr()).back = Some(new); (*new.as_ptr()).front = Some(old); } else { // If there's no back, then we're the empty list and need // to set the front too. self.front = Some(new); } // These things always happen! self.back = Some(new); self.len += 1; } } pub fn pop_front(&mut self) -> Option<T> { unsafe { // Only have to do stuff if there is a front node to pop. self.front.map(|node| { // Bring the Box back to life so we can move out its value and // Drop it (Box continues to magically understand this for us). let boxed_node = Box::from_raw(node.as_ptr()); let result = boxed_node.elem; // Make the next node into the new front. self.front = boxed_node.back; if let Some(new) = self.front { // Cleanup its reference to the removed node (*new.as_ptr()).front = None; } else { // If the front is now null, then this list is now empty! self.back = None; } self.len -= 1; result // Box gets implicitly freed here, knows there is no T. }) } } pub fn pop_back(&mut self) -> Option<T> { unsafe { // Only have to do stuff if there is a back node to pop. self.back.map(|node| { // Bring the Box front to life so we can move out its value and // Drop it (Box continues to magically understand this for us). let boxed_node = Box::from_raw(node.as_ptr()); let result = boxed_node.elem; // Make the next node into the new back. self.back = boxed_node.front; if let Some(new) = self.back { // Cleanup its reference to the removed node (*new.as_ptr()).back = None; } else { // If the back is now null, then this list is now empty! self.front = None; } self.len -= 1; result // Box gets implicitly freed here, knows there is no T. }) } } pub fn front(&self) -> Option<&T> { unsafe { self.front.map(|node| &(*node.as_ptr()).elem) } } pub fn front_mut(&mut self) -> Option<&mut T> { unsafe { self.front.map(|node| &mut (*node.as_ptr()).elem) } } pub fn back(&self) -> Option<&T> { unsafe { self.back.map(|node| &(*node.as_ptr()).elem) } } pub fn back_mut(&mut self) -> Option<&mut T> { unsafe { self.back.map(|node| &mut (*node.as_ptr()).elem) } } pub fn len(&self) -> usize { self.len } pub fn is_empty(&self) -> bool { self.len == 0 } pub fn clear(&mut self) { // Oh look it's drop again while self.pop_front().is_some() {} } pub fn iter(&self) -> Iter<T> { Iter { front: self.front, back: self.back, len: self.len, _boo: PhantomData, } } pub fn iter_mut(&mut self) -> IterMut<T> { IterMut { front: self.front, back: self.back, len: self.len, _boo: PhantomData, } } pub fn cursor_mut(&mut self) -> CursorMut<T> { CursorMut { list: self, cur: None, index: None, } } } impl<T> Drop for LinkedList<T> { fn drop(&mut self) { // Pop until we have to stop while self.pop_front().is_some() {} } } impl<T> Default for LinkedList<T> { fn default() -> Self { Self::new() } } impl<T: Clone> Clone for LinkedList<T> { fn clone(&self) -> Self { let mut new_list = Self::new(); for item in self { new_list.push_back(item.clone()); } new_list } } impl<T> Extend<T> for LinkedList<T> { fn extend<I: IntoIterator<Item = T>>(&mut self, iter: I) { for item in iter { self.push_back(item); } } } impl<T> FromIterator<T> for LinkedList<T> { fn from_iter<I: IntoIterator<Item = T>>(iter: I) -> Self { let mut list = Self::new(); list.extend(iter); list } } impl<T: Debug> Debug for LinkedList<T> { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.debug_list().entries(self).finish() } } impl<T: PartialEq> PartialEq for LinkedList<T> { fn eq(&self, other: &Self) -> bool { self.len() == other.len() && self.iter().eq(other) } } impl<T: Eq> Eq for LinkedList<T> {} impl<T: PartialOrd> PartialOrd for LinkedList<T> { fn partial_cmp(&self, other: &Self) -> Option<Ordering> { self.iter().partial_cmp(other) } } impl<T: Ord> Ord for LinkedList<T> { fn cmp(&self, other: &Self) -> Ordering { self.iter().cmp(other) } } impl<T: Hash> Hash for LinkedList<T> { fn hash<H: Hasher>(&self, state: &mut H) { self.len().hash(state); for item in self { item.hash(state); } } } impl<'a, T> IntoIterator for &'a LinkedList<T> { type IntoIter = Iter<'a, T>; type Item = &'a T; fn into_iter(self) -> Self::IntoIter { self.iter() } } impl<'a, T> Iterator for Iter<'a, T> { type Item = &'a T; fn next(&mut self) -> Option<Self::Item> { // While self.front == self.back is a tempting condition to check here, // it won't do the right for yielding the last element! That sort of // thing only works for arrays because of "one-past-the-end" pointers. if self.len > 0 { // We could unwrap front, but this is safer and easier self.front.map(|node| unsafe { self.len -= 1; self.front = (*node.as_ptr()).back; &(*node.as_ptr()).elem }) } else { None } } fn size_hint(&self) -> (usize, Option<usize>) { (self.len, Some(self.len)) } } impl<'a, T> DoubleEndedIterator for Iter<'a, T> { fn next_back(&mut self) -> Option<Self::Item> { if self.len > 0 { self.back.map(|node| unsafe { self.len -= 1; self.back = (*node.as_ptr()).front; &(*node.as_ptr()).elem }) } else { None } } } impl<'a, T> ExactSizeIterator for Iter<'a, T> { fn len(&self) -> usize { self.len } } impl<'a, T> IntoIterator for &'a mut LinkedList<T> { type IntoIter = IterMut<'a, T>; type Item = &'a mut T; fn into_iter(self) -> Self::IntoIter { self.iter_mut() } } impl<'a, T> Iterator for IterMut<'a, T> { type Item = &'a mut T; fn next(&mut self) -> Option<Self::Item> { // While self.front == self.back is a tempting condition to check here, // it won't do the right for yielding the last element! That sort of // thing only works for arrays because of "one-past-the-end" pointers. if self.len > 0 { // We could unwrap front, but this is safer and easier self.front.map(|node| unsafe { self.len -= 1; self.front = (*node.as_ptr()).back; &mut (*node.as_ptr()).elem }) } else { None } } fn size_hint(&self) -> (usize, Option<usize>) { (self.len, Some(self.len)) } } impl<'a, T> DoubleEndedIterator for IterMut<'a, T> { fn next_back(&mut self) -> Option<Self::Item> { if self.len > 0 { self.back.map(|node| unsafe { self.len -= 1; self.back = (*node.as_ptr()).front; &mut (*node.as_ptr()).elem }) } else { None } } } impl<'a, T> ExactSizeIterator for IterMut<'a, T> { fn len(&self) -> usize { self.len } } impl<T> IntoIterator for LinkedList<T> { type IntoIter = IntoIter<T>; type Item = T; fn into_iter(self) -> Self::IntoIter { IntoIter { list: self } } } impl<T> Iterator for IntoIter<T> { type Item = T; fn next(&mut self) -> Option<Self::Item> { self.list.pop_front() } fn size_hint(&self) -> (usize, Option<usize>) { (self.list.len, Some(self.list.len)) } } impl<T> DoubleEndedIterator for IntoIter<T> { fn next_back(&mut self) -> Option<Self::Item> { self.list.pop_back() } } impl<T> ExactSizeIterator for IntoIter<T> { fn len(&self) -> usize { self.list.len } } impl<'a, T> CursorMut<'a, T> { pub fn index(&self) -> Option<usize> { self.index } pub fn move_next(&mut self) { if let Some(cur) = self.cur { unsafe { // We're on a real element, go to its next (back) self.cur = (*cur.as_ptr()).back; if self.cur.is_some() { *self.index.as_mut().unwrap() += 1; } else { // We just walked to the ghost, no more index self.index = None; } } } else if !self.list.is_empty() { // We're at the ghost, and there is a real front, so move to it! self.cur = self.list.front; self.index = Some(0) } else { // We're at the ghost, but that's the only element... do nothing. } } pub fn move_prev(&mut self) { if let Some(cur) = self.cur { unsafe { // We're on a real element, go to its previous (front) self.cur = (*cur.as_ptr()).front; if self.cur.is_some() { *self.index.as_mut().unwrap() -= 1; } else { // We just walked to the ghost, no more index self.index = None; } } } else if !self.list.is_empty() { // We're at the ghost, and there is a real back, so move to it! self.cur = self.list.back; self.index = Some(self.list.len - 1) } else { // We're at the ghost, but that's the only element... do nothing. } } pub fn current(&mut self) -> Option<&mut T> { unsafe { self.cur.map(|node| &mut (*node.as_ptr()).elem) } } pub fn peek_next(&mut self) -> Option<&mut T> { unsafe { let next = if let Some(cur) = self.cur { // Normal case, try to follow the cur node's back pointer (*cur.as_ptr()).back } else { // Ghost case, try to use the list's front pointer self.list.front }; // Yield the element if the next node exists next.map(|node| &mut (*node.as_ptr()).elem) } } pub fn peek_prev(&mut self) -> Option<&mut T> { unsafe { let prev = if let Some(cur) = self.cur { // Normal case, try to follow the cur node's front pointer (*cur.as_ptr()).front } else { // Ghost case, try to use the list's back pointer self.list.back }; // Yield the element if the prev node exists prev.map(|node| &mut (*node.as_ptr()).elem) } } pub fn split_before(&mut self) -> LinkedList<T> { // We have this: // // list.front -> A <-> B <-> C <-> D <- list.back // ^ // cur // // // And we want to produce this: // // list.front -> C <-> D <- list.back // ^ // cur // // // return.front -> A <-> B <- return.back // if let Some(cur) = self.cur { // We are pointing at a real element, so the list is non-empty. unsafe { // Current state let old_len = self.list.len; let old_idx = self.index.unwrap(); let prev = (*cur.as_ptr()).front; // What self will become let new_len = old_len - old_idx; let new_front = self.cur; let new_back = self.list.back; let new_idx = Some(0); // What the output will become let output_len = old_len - new_len; let output_front = self.list.front; let output_back = prev; // Break the links between cur and prev if let Some(prev) = prev { (*cur.as_ptr()).front = None; (*prev.as_ptr()).back = None; } // Produce the result: self.list.len = new_len; self.list.front = new_front; self.list.back = new_back; self.index = new_idx; LinkedList { front: output_front, back: output_back, len: output_len, _boo: PhantomData, } } } else { // We're at the ghost, just replace our list with an empty one. // No other state needs to be changed. std::mem::replace(self.list, LinkedList::new()) } } pub fn split_after(&mut self) -> LinkedList<T> { // We have this: // // list.front -> A <-> B <-> C <-> D <- list.back // ^ // cur // // // And we want to produce this: // // list.front -> A <-> B <- list.back // ^ // cur // // // return.front -> C <-> D <- return.back // if let Some(cur) = self.cur { // We are pointing at a real element, so the list is non-empty. unsafe { // Current state let old_len = self.list.len; let old_idx = self.index.unwrap(); let next = (*cur.as_ptr()).back; // What self will become let new_len = old_idx + 1; let new_back = self.cur; let new_front = self.list.front; let new_idx = Some(old_idx); // What the output will become let output_len = old_len - new_len; let output_front = next; let output_back = self.list.back; // Break the links between cur and next if let Some(next) = next { (*cur.as_ptr()).back = None; (*next.as_ptr()).front = None; } // Produce the result: self.list.len = new_len; self.list.front = new_front; self.list.back = new_back; self.index = new_idx; LinkedList { front: output_front, back: output_back, len: output_len, _boo: PhantomData, } } } else { // We're at the ghost, just replace our list with an empty one. // No other state needs to be changed. std::mem::replace(self.list, LinkedList::new()) } } pub fn splice_before(&mut self, mut input: LinkedList<T>) { // We have this: // // input.front -> 1 <-> 2 <- input.back // // list.front -> A <-> B <-> C <- list.back // ^ // cur // // // Becoming this: // // list.front -> A <-> 1 <-> 2 <-> B <-> C <- list.back // ^ // cur // unsafe { // We can either `take` the input's pointers or `mem::forget` // it. Using `take` is more responsible in case we ever do custom // allocators or something that also needs to be cleaned up! if input.is_empty() { // Input is empty, do nothing. } else if let Some(cur) = self.cur { // Both lists are non-empty let in_front = input.front.take().unwrap(); let in_back = input.back.take().unwrap(); if let Some(prev) = (*cur.as_ptr()).front { // General Case, no boundaries, just internal fixups (*prev.as_ptr()).back = Some(in_front); (*in_front.as_ptr()).front = Some(prev); (*cur.as_ptr()).front = Some(in_back); (*in_back.as_ptr()).back = Some(cur); } else { // No prev, we're appending to the front (*cur.as_ptr()).front = Some(in_back); (*in_back.as_ptr()).back = Some(cur); self.list.front = Some(in_front); } // Index moves forward by input length *self.index.as_mut().unwrap() += input.len; } else if let Some(back) = self.list.back { // We're on the ghost but non-empty, append to the back let in_front = input.front.take().unwrap(); let in_back = input.back.take().unwrap(); (*back.as_ptr()).back = Some(in_front); (*in_front.as_ptr()).front = Some(back); self.list.back = Some(in_back); } else { // We're empty, become the input, remain on the ghost std::mem::swap(self.list, &mut input); } self.list.len += input.len; // Not necessary but Polite To Do input.len = 0; // Input dropped here } } pub fn splice_after(&mut self, mut input: LinkedList<T>) { // We have this: // // input.front -> 1 <-> 2 <- input.back // // list.front -> A <-> B <-> C <- list.back // ^ // cur // // // Becoming this: // // list.front -> A <-> B <-> 1 <-> 2 <-> C <- list.back // ^ // cur // unsafe { // We can either `take` the input's pointers or `mem::forget` // it. Using `take` is more responsible in case we ever do custom // allocators or something that also needs to be cleaned up! if input.is_empty() { // Input is empty, do nothing. } else if let Some(cur) = self.cur { // Both lists are non-empty let in_front = input.front.take().unwrap(); let in_back = input.back.take().unwrap(); if let Some(next) = (*cur.as_ptr()).back { // General Case, no boundaries, just internal fixups (*next.as_ptr()).front = Some(in_back); (*in_back.as_ptr()).back = Some(next); (*cur.as_ptr()).back = Some(in_front); (*in_front.as_ptr()).front = Some(cur); } else { // No next, we're appending to the back (*cur.as_ptr()).back = Some(in_front); (*in_front.as_ptr()).front = Some(cur); self.list.back = Some(in_back); } // Index doesn't change } else if let Some(front) = self.list.front { // We're on the ghost but non-empty, append to the front let in_front = input.front.take().unwrap(); let in_back = input.back.take().unwrap(); (*front.as_ptr()).front = Some(in_back); (*in_back.as_ptr()).back = Some(front); self.list.front = Some(in_front); } else { // We're empty, become the input, remain on the ghost std::mem::swap(self.list, &mut input); } self.list.len += input.len; // Not necessary but Polite To Do input.len = 0; // Input dropped here } } } unsafe impl<T: Send> Send for LinkedList<T> {} unsafe impl<T: Sync> Sync for LinkedList<T> {} unsafe impl<'a, T: Send> Send for Iter<'a, T> {} unsafe impl<'a, T: Sync> Sync for Iter<'a, T> {} unsafe impl<'a, T: Send> Send for IterMut<'a, T> {} unsafe impl<'a, T: Sync> Sync for IterMut<'a, T> {} #[allow(dead_code)] fn assert_properties() { fn is_send<T: Send>() {} fn is_sync<T: Sync>() {} is_send::<LinkedList<i32>>(); is_sync::<LinkedList<i32>>(); is_send::<IntoIter<i32>>(); is_sync::<IntoIter<i32>>(); is_send::<Iter<i32>>(); is_sync::<Iter<i32>>(); is_send::<IterMut<i32>>(); is_sync::<IterMut<i32>>(); fn linked_list_covariant<'a, T>(x: LinkedList<&'static T>) -> LinkedList<&'a T> { x } fn iter_covariant<'i, 'a, T>(x: Iter<'i, &'static T>) -> Iter<'i, &'a T> { x } fn into_iter_covariant<'a, T>(x: IntoIter<&'static T>) -> IntoIter<&'a T> { x } /// ```compile_fail,E0308 /// use linked_list::IterMut; /// /// fn iter_mut_covariant<'i, 'a, T>(x: IterMut<'i, &'static T>) -> IterMut<'i, &'a T> { x } /// ``` fn iter_mut_invariant() {} } #[cfg(test)] mod test { use super::LinkedList; fn generate_test() -> LinkedList<i32> { list_from(&[0, 1, 2, 3, 4, 5, 6]) } fn list_from<T: Clone>(v: &[T]) -> LinkedList<T> { v.iter().map(|x| (*x).clone()).collect() } #[test] fn test_basic_front() { let mut list = LinkedList::new(); // Try to break an empty list assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); // Try to break a one item list list.push_front(10); assert_eq!(list.len(), 1); assert_eq!(list.pop_front(), Some(10)); assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); // Mess around list.push_front(10); assert_eq!(list.len(), 1); list.push_front(20); assert_eq!(list.len(), 2); list.push_front(30); assert_eq!(list.len(), 3); assert_eq!(list.pop_front(), Some(30)); assert_eq!(list.len(), 2); list.push_front(40); assert_eq!(list.len(), 3); assert_eq!(list.pop_front(), Some(40)); assert_eq!(list.len(), 2); assert_eq!(list.pop_front(), Some(20)); assert_eq!(list.len(), 1); assert_eq!(list.pop_front(), Some(10)); assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); assert_eq!(list.pop_front(), None); assert_eq!(list.len(), 0); } #[test] fn test_basic() { let mut m = LinkedList::new(); assert_eq!(m.pop_front(), None); assert_eq!(m.pop_back(), None); assert_eq!(m.pop_front(), None); m.push_front(1); assert_eq!(m.pop_front(), Some(1)); m.push_back(2); m.push_back(3); assert_eq!(m.len(), 2); assert_eq!(m.pop_front(), Some(2)); assert_eq!(m.pop_front(), Some(3)); assert_eq!(m.len(), 0); assert_eq!(m.pop_front(), None); m.push_back(1); m.push_back(3); m.push_back(5); m.push_back(7); assert_eq!(m.pop_front(), Some(1)); let mut n = LinkedList::new(); n.push_front(2); n.push_front(3); { assert_eq!(n.front().unwrap(), &3); let x = n.front_mut().unwrap(); assert_eq!(*x, 3); *x = 0; } { assert_eq!(n.back().unwrap(), &2); let y = n.back_mut().unwrap(); assert_eq!(*y, 2); *y = 1; } assert_eq!(n.pop_front(), Some(0)); assert_eq!(n.pop_front(), Some(1)); } #[test] fn test_iterator() { let m = generate_test(); for (i, elt) in m.iter().enumerate() { assert_eq!(i as i32, *elt); } let mut n = LinkedList::new(); assert_eq!(n.iter().next(), None); n.push_front(4); let mut it = n.iter(); assert_eq!(it.size_hint(), (1, Some(1))); assert_eq!(it.next().unwrap(), &4); assert_eq!(it.size_hint(), (0, Some(0))); assert_eq!(it.next(), None); } #[test] fn test_iterator_double_end() { let mut n = LinkedList::new(); assert_eq!(n.iter().next(), None); n.push_front(4); n.push_front(5); n.push_front(6); let mut it = n.iter(); assert_eq!(it.size_hint(), (3, Some(3))); assert_eq!(it.next().unwrap(), &6); assert_eq!(it.size_hint(), (2, Some(2))); assert_eq!(it.next_back().unwrap(), &4); assert_eq!(it.size_hint(), (1, Some(1))); assert_eq!(it.next_back().unwrap(), &5); assert_eq!(it.next_back(), None); assert_eq!(it.next(), None); } #[test] fn test_rev_iter() { let m = generate_test(); for (i, elt) in m.iter().rev().enumerate() { assert_eq!(6 - i as i32, *elt); } let mut n = LinkedList::new(); assert_eq!(n.iter().rev().next(), None); n.push_front(4); let mut it = n.iter().rev(); assert_eq!(it.size_hint(), (1, Some(1))); assert_eq!(it.next().unwrap(), &4); assert_eq!(it.size_hint(), (0, Some(0))); assert_eq!(it.next(), None); } #[test] fn test_mut_iter() { let mut m = generate_test(); let mut len = m.len(); for (i, elt) in m.iter_mut().enumerate() { assert_eq!(i as i32, *elt); len -= 1; } assert_eq!(len, 0); let mut n = LinkedList::new(); assert!(n.iter_mut().next().is_none()); n.push_front(4); n.push_back(5); let mut it = n.iter_mut(); assert_eq!(it.size_hint(), (2, Some(2))); assert!(it.next().is_some()); assert!(it.next().is_some()); assert_eq!(it.size_hint(), (0, Some(0))); assert!(it.next().is_none()); } #[test] fn test_iterator_mut_double_end() { let mut n = LinkedList::new(); assert!(n.iter_mut().next_back().is_none()); n.push_front(4); n.push_front(5); n.push_front(6); let mut it = n.iter_mut(); assert_eq!(it.size_hint(), (3, Some(3))); assert_eq!(*it.next().unwrap(), 6); assert_eq!(it.size_hint(), (2, Some(2))); assert_eq!(*it.next_back().unwrap(), 4); assert_eq!(it.size_hint(), (1, Some(1))); assert_eq!(*it.next_back().unwrap(), 5); assert!(it.next_back().is_none()); assert!(it.next().is_none()); } #[test] fn test_eq() { let mut n: LinkedList<u8> = list_from(&[]); let mut m = list_from(&[]); assert!(n == m); n.push_front(1); assert!(n != m); m.push_back(1); assert!(n == m); let n = list_from(&[2, 3, 4]); let m = list_from(&[1, 2, 3]); assert!(n != m); } #[test] fn test_ord() { let n = list_from(&[]); let m = list_from(&[1, 2, 3]); assert!(n < m); assert!(m > n); assert!(n <= n); assert!(n >= n); } #[test] fn test_ord_nan() { let nan = 0.0f64 / 0.0; let n = list_from(&[nan]); let m = list_from(&[nan]); assert!(!(n < m)); assert!(!(n > m)); assert!(!(n <= m)); assert!(!(n >= m)); let n = list_from(&[nan]); let one = list_from(&[1.0f64]); assert!(!(n < one)); assert!(!(n > one)); assert!(!(n <= one)); assert!(!(n >= one)); let u = list_from(&[1.0f64, 2.0, nan]); let v = list_from(&[1.0f64, 2.0, 3.0]); assert!(!(u < v)); assert!(!(u > v)); assert!(!(u <= v)); assert!(!(u >= v)); let s = list_from(&[1.0f64, 2.0, 4.0, 2.0]); let t = list_from(&[1.0f64, 2.0, 3.0, 2.0]); assert!(!(s < t)); assert!(s > one); assert!(!(s <= one)); assert!(s >= one); } #[test] fn test_debug() { let list: LinkedList<i32> = (0..10).collect(); assert_eq!(format!("{:?}", list), "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"); let list: LinkedList<&str> = vec!["just", "one", "test", "more"] .iter() .copied() .collect(); assert_eq!(format!("{:?}", list), r#"["just", "one", "test", "more"]"#); } #[test] fn test_hashmap() { // Check that HashMap works with this as a key let list1: LinkedList<i32> = (0..10).collect(); let list2: LinkedList<i32> = (1..11).collect(); let mut map = std::collections::HashMap::new(); assert_eq!(map.insert(list1.clone(), "list1"), None); assert_eq!(map.insert(list2.clone(), "list2"), None); assert_eq!(map.len(), 2); assert_eq!(map.get(&list1), Some(&"list1")); assert_eq!(map.get(&list2), Some(&"list2")); assert_eq!(map.remove(&list1), Some("list1")); assert_eq!(map.remove(&list2), Some("list2")); assert!(map.is_empty()); } #[test] fn test_cursor_move_peek() { let mut m: LinkedList<u32> = LinkedList::new(); m.extend([1, 2, 3, 4, 5, 6]); let mut cursor = m.cursor_mut(); cursor.move_next(); assert_eq!(cursor.current(), Some(&mut 1)); assert_eq!(cursor.peek_next(), Some(&mut 2)); assert_eq!(cursor.peek_prev(), None); assert_eq!(cursor.index(), Some(0)); cursor.move_prev(); assert_eq!(cursor.current(), None); assert_eq!(cursor.peek_next(), Some(&mut 1)); assert_eq!(cursor.peek_prev(), Some(&mut 6)); assert_eq!(cursor.index(), None); cursor.move_next(); cursor.move_next(); assert_eq!(cursor.current(), Some(&mut 2)); assert_eq!(cursor.peek_next(), Some(&mut 3)); assert_eq!(cursor.peek_prev(), Some(&mut 1)); assert_eq!(cursor.index(), Some(1)); let mut cursor = m.cursor_mut(); cursor.move_prev(); assert_eq!(cursor.current(), Some(&mut 6)); assert_eq!(cursor.peek_next(), None); assert_eq!(cursor.peek_prev(), Some(&mut 5)); assert_eq!(cursor.index(), Some(5)); cursor.move_next(); assert_eq!(cursor.current(), None); assert_eq!(cursor.peek_next(), Some(&mut 1)); assert_eq!(cursor.peek_prev(), Some(&mut 6)); assert_eq!(cursor.index(), None); cursor.move_prev(); cursor.move_prev(); assert_eq!(cursor.current(), Some(&mut 5)); assert_eq!(cursor.peek_next(), Some(&mut 6)); assert_eq!(cursor.peek_prev(), Some(&mut 4)); assert_eq!(cursor.index(), Some(4)); } #[test] fn test_cursor_mut_insert() { let mut m: LinkedList<u32> = LinkedList::new(); m.extend([1, 2, 3, 4, 5, 6]); let mut cursor = m.cursor_mut(); cursor.move_next(); cursor.splice_before(Some(7).into_iter().collect()); cursor.splice_after(Some(8).into_iter().collect()); // check_links(&m); assert_eq!( m.iter().cloned().collect::<Vec<_>>(), &[7, 1, 8, 2, 3, 4, 5, 6] ); let mut cursor = m.cursor_mut(); cursor.move_next(); cursor.move_prev(); cursor.splice_before(Some(9).into_iter().collect()); cursor.splice_after(Some(10).into_iter().collect()); check_links(&m); assert_eq!( m.iter().cloned().collect::<Vec<_>>(), &[10, 7, 1, 8, 2, 3, 4, 5, 6, 9] ); /* remove_current not impl'd let mut cursor = m.cursor_mut(); cursor.move_next(); cursor.move_prev(); assert_eq!(cursor.remove_current(), None); cursor.move_next(); cursor.move_next(); assert_eq!(cursor.remove_current(), Some(7)); cursor.move_prev(); cursor.move_prev(); cursor.move_prev(); assert_eq!(cursor.remove_current(), Some(9)); cursor.move_next(); assert_eq!(cursor.remove_current(), Some(10)); check_links(&m); assert_eq!(m.iter().cloned().collect::<Vec<_>>(), &[1, 8, 2, 3, 4, 5, 6]); */ let mut m: LinkedList<u32> = LinkedList::new(); m.extend([1, 8, 2, 3, 4, 5, 6]); let mut cursor = m.cursor_mut(); cursor.move_next(); let mut p: LinkedList<u32> = LinkedList::new(); p.extend([100, 101, 102, 103]); let mut q: LinkedList<u32> = LinkedList::new(); q.extend([200, 201, 202, 203]); cursor.splice_after(p); cursor.splice_before(q); check_links(&m); assert_eq!( m.iter().cloned().collect::<Vec<_>>(), &[200, 201, 202, 203, 1, 100, 101, 102, 103, 8, 2, 3, 4, 5, 6] ); let mut cursor = m.cursor_mut(); cursor.move_next(); cursor.move_prev(); let tmp = cursor.split_before(); assert_eq!(m.into_iter().collect::<Vec<_>>(), &[]); m = tmp; let mut cursor = m.cursor_mut(); cursor.move_next(); cursor.move_next(); cursor.move_next(); cursor.move_next(); cursor.move_next(); cursor.move_next(); cursor.move_next(); let tmp = cursor.split_after(); assert_eq!( tmp.into_iter().collect::<Vec<_>>(), &[102, 103, 8, 2, 3, 4, 5, 6] ); check_links(&m); assert_eq!( m.iter().cloned().collect::<Vec<_>>(), &[200, 201, 202, 203, 1, 100, 101] ); } fn check_links<T: Eq + std::fmt::Debug>(list: &LinkedList<T>) { let from_front: Vec<_> = list.iter().collect(); let from_back: Vec<_> = list.iter().rev().collect(); let re_reved: Vec<_> = from_back.into_iter().rev().collect(); assert_eq!(from_front, re_reved); } } }
A Bunch of Silly Lists
Alright. That's it. We made all the lists.
ahahahaha
No
There's always more lists.
This chapter is a living document of the more ridiculous linked lists and how they interact with Rust.
- The Double Single
- The Stack Allocated List
- The Self-Referential Arena List?
- The GhostCell List?
The Double Singly-Linked List
We struggled with doubly-linked lists because they have tangled ownership semantics: no node strictly owns any other node. However we struggled with this because we brought in our preconceived notions of what a linked list is. Namely, we assumed that all the links go in the same direction.
Instead, we can smash our list into two halves: one going to the left, and one going to the right:
// lib.rs
// ...
pub mod silly1; // NEW!
// silly1.rs
use crate::second::List as Stack;
struct List<T> {
left: Stack<T>,
right: Stack<T>,
}
Now, rather than having a mere safe stack, we have a general purpose list. We can grow the list leftwards or rightwards by pushing onto either stack. We can also "walk" along the list by popping values off one end and onto the other. To avoid needless allocations, we're going to copy the source of our safe Stack to get access to its private details:
pub struct Stack<T> {
head: Link<T>,
}
type Link<T> = Option<Box<Node<T>>>;
struct Node<T> {
elem: T,
next: Link<T>,
}
impl<T> Stack<T> {
pub fn new() -> Self {
Stack { head: None }
}
pub fn push(&mut self, elem: T) {
let new_node = Box::new(Node {
elem: elem,
next: self.head.take(),
});
self.head = Some(new_node);
}
pub fn pop(&mut self) -> Option<T> {
self.head.take().map(|node| {
let node = *node;
self.head = node.next;
node.elem
})
}
pub fn peek(&self) -> Option<&T> {
self.head.as_ref().map(|node| {
&node.elem
})
}
pub fn peek_mut(&mut self) -> Option<&mut T> {
self.head.as_mut().map(|node| {
&mut node.elem
})
}
}
impl<T> Drop for Stack<T> {
fn drop(&mut self) {
let mut cur_link = self.head.take();
while let Some(mut boxed_node) = cur_link {
cur_link = boxed_node.next.take();
}
}
}
And just rework push and pop a bit:
pub fn push(&mut self, elem: T) {
let new_node = Box::new(Node {
elem: elem,
next: None,
});
self.push_node(new_node);
}
fn push_node(&mut self, mut node: Box<Node<T>>) {
node.next = self.head.take();
self.head = Some(node);
}
pub fn pop(&mut self) -> Option<T> {
self.pop_node().map(|node| {
node.elem
})
}
fn pop_node(&mut self) -> Option<Box<Node<T>>> {
self.head.take().map(|mut node| {
self.head = node.next.take();
node
})
}
Now we can make our List:
pub struct List<T> {
left: Stack<T>,
right: Stack<T>,
}
impl<T> List<T> {
fn new() -> Self {
List { left: Stack::new(), right: Stack::new() }
}
}
And we can do the usual stuff:
pub fn push_left(&mut self, elem: T) { self.left.push(elem) }
pub fn push_right(&mut self, elem: T) { self.right.push(elem) }
pub fn pop_left(&mut self) -> Option<T> { self.left.pop() }
pub fn pop_right(&mut self) -> Option<T> { self.right.pop() }
pub fn peek_left(&self) -> Option<&T> { self.left.peek() }
pub fn peek_right(&self) -> Option<&T> { self.right.peek() }
pub fn peek_left_mut(&mut self) -> Option<&mut T> { self.left.peek_mut() }
pub fn peek_right_mut(&mut self) -> Option<&mut T> { self.right.peek_mut() }
But most interestingly, we can walk around!
pub fn go_left(&mut self) -> bool {
self.left.pop_node().map(|node| {
self.right.push_node(node);
}).is_some()
}
pub fn go_right(&mut self) -> bool {
self.right.pop_node().map(|node| {
self.left.push_node(node);
}).is_some()
}
We return booleans here as just a convenience to indicate whether we actually managed to move. Now let's test this baby out:
#[cfg(test)]
mod test {
use super::List;
#[test]
fn walk_aboot() {
let mut list = List::new(); // [_]
list.push_left(0); // [0,_]
list.push_right(1); // [0, _, 1]
assert_eq!(list.peek_left(), Some(&0));
assert_eq!(list.peek_right(), Some(&1));
list.push_left(2); // [0, 2, _, 1]
list.push_left(3); // [0, 2, 3, _, 1]
list.push_right(4); // [0, 2, 3, _, 4, 1]
while list.go_left() {} // [_, 0, 2, 3, 4, 1]
assert_eq!(list.pop_left(), None);
assert_eq!(list.pop_right(), Some(0)); // [_, 2, 3, 4, 1]
assert_eq!(list.pop_right(), Some(2)); // [_, 3, 4, 1]
list.push_left(5); // [5, _, 3, 4, 1]
assert_eq!(list.pop_right(), Some(3)); // [5, _, 4, 1]
assert_eq!(list.pop_left(), Some(5)); // [_, 4, 1]
assert_eq!(list.pop_right(), Some(4)); // [_, 1]
assert_eq!(list.pop_right(), Some(1)); // [_]
assert_eq!(list.pop_right(), None);
assert_eq!(list.pop_left(), None);
}
}
> cargo test
Running target/debug/lists-5c71138492ad4b4a
running 16 tests
test fifth::test::into_iter ... ok
test fifth::test::basics ... ok
test fifth::test::iter ... ok
test fifth::test::iter_mut ... ok
test fourth::test::into_iter ... ok
test fourth::test::basics ... ok
test fourth::test::peek ... ok
test first::test::basics ... ok
test second::test::into_iter ... ok
test second::test::basics ... ok
test second::test::iter ... ok
test second::test::iter_mut ... ok
test third::test::basics ... ok
test third::test::iter ... ok
test second::test::peek ... ok
test silly1::test::walk_aboot ... ok
test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured
This is an extreme example of a finger data structure, where we maintain some kind of finger into the structure, and as a consequence can support operations on locations in time proportional to the distance from the finger.
We can make very fast changes to the list around our finger, but if we want
to make changes far away from our finger we have to walk all the way over there.
We can permanently walk over there by shifting the elements from one stack to
the other, or we could just walk along the links with an &mut
temporarily to do the changes. However the &mut can never go back up the
list, while our finger can!
The Stack-Allocated Linked List
This book is largely focused on heap-allocated linked lists, because those are the most common and practical, but we don't have to use heap allocation. Heap allocation is nice because it makes it easy to dynamically allocate memory. Stack allocation is less friendly in this regard — things like C's alloca are widely regarded as Very Cursed And Problematic.
So let's allocate memory on the stack the easy way: by calling a function and getting a new stack frame with more space! This is a very silly solution to our problem but also genuinely practical and useful. It's done all the time, potentially without actually even thinking about it as a linked list!
Any time you're doing something recursively, you can just pass a pointer to the current step's state to the next step. If that pointer itself is part of the state, then you've created a linked list that's stack-allocated!
Now of course we're in the silly part of the book so we're going to do this in a silly way: by making the linked list the star and forcing all the user's code to live in a swamp of callbacks. Everybody loves nested callbacks!
Our List type will just be a Node with a reference to another Node:
#![allow(unused)] fn main() { pub struct List<'a, T> { pub data: T, pub prev: Option<&'a List<'a, T>>, } }
And it will have only one operation, push, which will take the old list, the state for the current node, and a callback. The new list will be produced in the callback. We will also let callbacks return any value, which push will return when it completes:
impl<'a, T> List<'a, T> {
pub fn push<U>(
prev: Option<&'a List<'a, T>>,
data: T,
callback: impl FnOnce(&List<'a, T>) -> U,
) -> U {
let list = List { data, prev };
callback(&list)
}
}
That's it! We can use it like this:
List::push(None, 3, |list| {
println!("{}", list.data);
List::push(Some(list), 5, |list| {
println!("{}", list.data);
List::push(Some(list), 13, |list| {
println!("{}", list.data);
})
})
})
It's beautiful. 😿
The user can already traverse this list by using while-let to walk over the prev values, but just for fun, let's implement an iterator, which is the usual:
impl<'a, T> List<'a, T> {
pub fn iter(&'a self) -> Iter<'a, T> {
Iter { next: Some(self) }
}
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&mut self) -> Option<Self::Item> {
self.next.map(|node| {
self.next = node.prev;
&node.data
})
}
}
Let's test it out:
#[cfg(test)]
mod test {
use super::List;
#[test]
fn elegance() {
List::push(None, 3, |list| {
assert_eq!(list.iter().copied().sum::<i32>(), 3);
List::push(Some(list), 5, |list| {
assert_eq!(list.iter().copied().sum::<i32>(), 5 + 3);
List::push(Some(list), 13, |list| {
assert_eq!(list.iter().copied().sum::<i32>(), 13 + 5 + 3);
})
})
})
}
}
> cargo test
running 18 tests
test fifth::test::into_iter ... ok
test fifth::test::iter ... ok
test fifth::test::iter_mut ... ok
test fifth::test::basics ... ok
test fifth::test::miri_food ... ok
test first::test::basics ... ok
test second::test::into_iter ... ok
test fourth::test::peek ... ok
test fourth::test::into_iter ... ok
test second::test::iter_mut ... ok
test fourth::test::basics ... ok
test second::test::basics ... ok
test second::test::iter ... ok
test third::test::basics ... ok
test silly1::test::walk_aboot ... ok
test silly2::test::elegance ... ok
test second::test::peek ... ok
test third::test::iter ... ok
test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out;
Now at this point you might be wonder "hey can I mutate the data stored in a node?". Maybe! Let's try to make the list use mutable references instead of shared ones:
#![allow(unused)] fn main() { pub struct List<'a, T> { pub data: T, pub prev: Option<&'a mut List<'a, T>>, } pub struct Iter<'a, T> { next: Option<&'a List<'a, T>>, } impl<'a, T> List<'a, T> { pub fn push<U>( prev: Option<&'a mut List<'a, T>>, data: T, callback: impl FnOnce(&mut List<'a, T>) -> U, ) -> U { let mut list = List { data, prev }; callback(&mut list) } pub fn iter(&'a self) -> Iter<'a, T> { Iter { next: Some(self) } } } impl<'a, T> Iterator for Iter<'a, T> { type Item = &'a T; fn next(&mut self) -> Option<Self::Item> { self.next.map(|node| { self.next = node.prev.as_ref().map(|prev| &**prev); &node.data }) } } }
> cargo test
error[E0521]: borrowed data escapes outside of closure
--> src\silly2.rs:47:32
|
46 | List::push(Some(list), 13, |list| {
| ----
| |
| `list` declared here, outside of the closure body
| `list` is a reference that is only valid in the closure body
47 | assert_eq!(list.iter().copied().sum::<i32>(), 13 + 5 + 3);
| ^^^^^^^^^^^ `list` escapes the closure body here
error[E0521]: borrowed data escapes outside of closure
--> src\silly2.rs:45:28
|
44 | List::push(Some(list), 5, |list| {
| ----
| |
| `list` declared here, outside of the closure body
| `list` is a reference that is only valid in the closure body
45 | assert_eq!(list.iter().copied().sum::<i32>(), 5 + 3);
| ^^^^^^^^^^^ `list` escapes the closure body here
<ad infinitum>
Whelp. Seems like it doesn't like our iterator. Maybe we messed that up? Let's simplify the test a bit to check:
#[test]
fn elegance() {
List::push(None, 3, |list| {
assert_eq!(list.data, 3);
List::push(Some(list), 5, |list| {
assert_eq!(list.data, 5);
List::push(Some(list), 13, |list| {
assert_eq!(list.data, 13);
})
})
})
}
> cargo test
error[E0521]: borrowed data escapes outside of closure
--> src\silly2.rs:46:17
|
44 | List::push(Some(list), 5, |list| {
| ----
| |
| `list` declared here, outside of the closure body
| `list` is a reference that is only valid in the closure body
45 | assert_eq!(list.data, 5);
46 | / List::push(Some(list), 13, |list| {
47 | | assert_eq!(list.data, 13);
48 | | })
| |______^ `list` escapes the closure body here
error[E0521]: borrowed data escapes outside of closure
--> src\silly2.rs:44:13
|
42 | List::push(None, 3, |list| {
| ----
| |
| `list` declared here, outside of the closure body
| `list` is a reference that is only valid in the closure body
43 | assert_eq!(list.data, 3);
44 | / List::push(Some(list), 5, |list| {
45 | | assert_eq!(list.data, 5);
46 | | List::push(Some(list), 13, |list| {
47 | | assert_eq!(list.data, 13);
48 | | })
49 | | })
| |______________^ `list` escapes the closure body here
Hmm no that's still some hot garbage.
The problem is that our list is accidentally(😉) relying on variance. Variance is a complicated subject but let's look at it in simplified terms here:
Each list contains a reference to a List with the exact same type as itself. From the perspective of the inner-most list, that means all lists are using the same lifetime as itself, but this is objectively false: each node in the list lives strictly longer than the next one, because they are literally in nested scopes!
So... why did the code compile when we were using shared references? Because in many cases, the compiler knows it's safe to have something that lives "too long"! When we stuff a reference to a list into the next one, the compiler is quietly "shrinking" down the lifetimes to make them fit what the new list expects. This lifetime shrinking is variance.
It's the exact same trick in languages with inheritance that let's you pass a Cat where an Animal (a supertype of a Cat) is expected. Intuitively we know it's fine to pass a Cat when an Animal is expected because a Cat is just an Animal and more. It's fine to forget the "and more" part for a while, right?
Similarly, a larger lifetime is just a smaller lifetime and more. So it's fine to forget the "and more" here too!
But of course you are now wondering: then why doesn't the mutable reference version work!?
Well, variance isn't always safe. If our code did compile, we could have written a use-after-free like this:
List::push(None, 3, |list| {
List::push(Some(list), 5, |list| {
List::push(Some(list), 13, |list| {
// HAHAHA all the lifetimes are the same, so the compiler will
// let me rewrite my parent to hold a mutable reference to myself!
// I will create all the use-after-frees!!
*list.prev.as_mut().unwrap().prev = Some(list);
})
})
})
The problem with forgetting details is that somewhere else might remember those details and expect them to remain true. That is a very big problem once you introduce mutation. If you're not careful, the code that doesn't remember the "and more" that we threw away might think it's fine to write things to places that "remember" and expect the "and more" to still be there.
Put in terms of inheritance: this code has to be illegal:
let mut my_kitty = Cat; // Make a Cat (long lifetime)
let animal: &mut Animal = &mut my_kitty; // Forget it's a Cat (shorten lifetime)
*animal = Dog; // Write a Dog (short lifetime)
my_kitty.meow(); // Meowing Dog! (Use After Free)
So while you can shorten the lifetime of a mutable reference, once you start nesting them things become "invariant" and you're not allowed to shorten lifetimes anymore.
Specifically &mut &'big mut T cannot be converted to &mut &'small mut T,
where 'big is bigger than 'small. Or more formally, &'a mut T is covariant
over 'a but invariant over T.
Fun fact: Java actually specifically lets you do this kind of thing, but it does runtime checks to prevent meowing dogs.
So what can we do to mutate the data? Use interior mutability! This lets us tell the compiler that we just want to be able to mutate the data but won't touch the references.
We can just revert back to the previous version of our code with shared references,
and use Cell in a new test:
#[test]
fn cell() {
use std::cell::Cell;
List::push(None, Cell::new(3), |list| {
List::push(Some(list), Cell::new(5), |list| {
List::push(Some(list), Cell::new(13), |list| {
// Multiply every value in the list by 10
for val in list.iter() {
val.set(val.get() * 10)
}
let mut vals = list.iter();
assert_eq!(vals.next().unwrap().get(), 130);
assert_eq!(vals.next().unwrap().get(), 50);
assert_eq!(vals.next().unwrap().get(), 30);
assert_eq!(vals.next(), None);
assert_eq!(vals.next(), None);
})
})
})
}
> cargo test
running 19 tests
test fifth::test::into_iter ... ok
test fifth::test::basics ... ok
test fifth::test::iter_mut ... ok
test fifth::test::iter ... ok
test fourth::test::basics ... ok
test fourth::test::into_iter ... ok
test second::test::into_iter ... ok
test first::test::basics ... ok
test fourth::test::peek ... ok
test second::test::basics ... ok
test fifth::test::miri_food ... ok
test silly2::test::cell ... ok
test third::test::iter ... ok
test second::test::iter_mut ... ok
test second::test::peek ... ok
test silly1::test::walk_aboot ... ok
test silly2::test::elegance ... ok
test third::test::basics ... ok
test second::test::iter ... ok
test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out;
Easy as recursive pie! ✨