The wiki of my highlights

I received my first Kindle as a Christmas present in 2013, and since then I have read most of the books I get through on this small device, which has followed me in my backpack everywhere, from mountain huts to the Amazon jungle. When I read on the Kindle, especially non-fiction (but not only), I often highlight passages that strike me as important, provocative, full of a richness I would like to remember. My Kindle holds more than three thousand highlighted sentences, in three languages and spread over a hundred and forty books, which I marked because they struck me at that moment and which, as happens to everyone, I never looked at again.

As I have already mentioned on these pages, for some time now I have been toying with the power of artificial intelligence tools. LLMs and agents have invaded everyone’s lives, but those of us researchers in particular, on the one hand frightening us with their effectiveness (will they replace us without mercy?) or making us feel omnipotent (Ah! Now we can do a thousand times more than we used to!), and on the other hand questioning us deeply about how our profession will change. If mathematicians were the first to take the AI tsunami in the face, physicists are right behind them, perhaps with the advantage that many of the things experimental physicists do are tools for some other end, and that AI replacing, speeding up or strengthening those tools is a good thing.

In this process of exploring the possibilities opened up by AI, this autumn I tried to do something with the highlights on my Kindle. I had no precise goal, more a vague idea of consolidating information that I hope to be able to use (also) for work. I had already done a similar exercise with the writings on this blog and my “founding principles”, remember? In the case of the Kindle highlights, I asked an LLM to turn them into a wiki that would be not so much a summary of the books as a map of what I had found in them. This post tells how it went, what works and above all what was not working without my noticing. Ideally it should be the first episode of a small series about my experiments with artificial intelligence, while the second will talk about a more serious wiki, the one I use for my work on CERN’s future collider.

The idea of a second brain, applied to my Kindle

The idea comes from Andrej Karpathy, a researcher at Anthropic, who a few months ago proposed building a “second brain” in the form of an LLM wiki, containing a distilled and organic form of the information we collect (for work, for pleasure, it hardly matters). Instead of querying every time the raw documents we have gathered, collected and digested on a given subject (because one assumes that people still read and internalise the sources of information they collect, which to me seems an essential step), the construction and maintenance of a wiki of linked pages is delegated to an AI model, and the wiki is then read and browsed to find information again, or to discover connections that might not have been clear on a first reading. The rules are few and simple: the raw data are never touched, the wiki belongs to the model, which writes and updates it, and the person who runs it decides what goes in and checks the result.

In the case of my Kindle highlights, this is the process I set up:

1. I connect the Kindle to the Mac and copy (automatically) the highlights file, updating it with respect to the previous pass (I usually do this when I finish reading a book).
2. A script splits the highlights by book, one text file each: these are the raw data.
3. An agent (Claude Code) reads the new files and writes the pages: one per book, one per author, and thematic pages such as faith, work, identity, mountains, parenting.
4. Everything lives in Obsidian, with the links between pages and the graph that shows them.

Books with few highlights get a short page, while for those I have highlighted the most the agent rereads the whole book from the EPUB I keep elsewhere in my Calibre library, and reconstructs, chapter by chapter, where the sentences I highlighted fall. As of today the wiki holds 141 books, about 3,300 highlights, 104 authors, 13 thematic pages and 5 cross-cutting analyses.

What came out of it that was good

Apart from having learned technical details and automation, the wiki that came out of it brought to light some interesting aspects of how I read, what I read, and what my highlights say about me. The most interesting part is in fact not the pages on individual books, but the “cross-cutting” ones that point out common themes, highlighted across many books, even apparently distant ones.

A recent example: after finishing Working Identity by Herminia Ibarra, a book on how people change course in mid-career, I asked for an analysis of what my library says about changing direction in the middle of a working life, and the agent, searching the highlights in English, Italian and French, brought together Ibarra, two books by James Hollis on the second half of life, Michael Lopp, Lori Gottlieb, Massimo Recalcati and others.

The most useful result, then, lies not where the books agree, but where they do not: for Ibarra, for example, the new self is built going forward, by acting and experimenting, while for Hollis it is recovered by listening to something that was there from the beginning, and Recalcati, who writes about grief and not about careers, raises an objection that the career books ignore entirely, namely that quickly replacing what has been lost can be a way of not working through the loss. None of these books, on its own, would have given me that tension, which instead emerged almost spontaneously by setting my highlights side by side.

The problems you don’t see

The wiki seemed in excellent health, with well-written pages, dense links and a last quality check, done by the same model, that reported no particularly blocking problem, until I looked a little more carefully. Some problems were technical, others more conceptual.

The check said everything was fine, but there were 44 “broken” links. That check had been done by the model rereading the pages, while a simple script that follows each link and verifies that the target page exists, not intelligent in the AI sense but deterministic like “classic” computer programs, found them in a second, confirming that mechanical things need a mechanical tool.

A quarter of the highlights were invisible to search. The wiki also has a local semantic search, where you type “career change” and get the sentences closest in meaning, in any language, and yet searching for exactly “career change” did not bring up Ibarra’s book. The reason was that the script reading the files expected an old format and silently skipped everything imported later, that is, about 830 highlights and 39 entire books, including the ones I had just read, without an error or a warning, only with somewhat worse results. A bit of cleaning up and restructuring solved the issue, but it reminded me that even a result that looks well polished and packaged by an AI can hide errors, and that only a check designed to find inconsistencies that are not immediately visible can solve the problem.

Books with a single sentence had five “key ideas”. For a novel from which I had highlighted a single line, the page listed five lessons from the book taken from a summary found online, well written and plausible, but completely disconnected from my reading, so that the wiki ended up describing the books, and not what I had found in them.

A few hallucinations

While rereading Le Royaume by Emmanuel Carrère, a book I finished recently, enjoyed a great deal and highlighted even more, the agent, updating the corresponding page, found a sentence attributed to Carrère that appeared neither in my highlights nor in the book, and on the same page also the wrong prize, the Renaudot, which Carrère did win, but for Limonov. An invented quotation, in a wiki built precisely to preserve quotations, is the kind of error you do not find by reading, because it sounds right, has the author’s style and sits in the right place; so I had a script written that, for every sentence in quotation marks in the wiki, looks for the exact text first among my highlights and then in the full text of the book.

Outcome	Quotations
Checked (excluding those under four words)	about 2,400
Correct from the start	about 2,100
Restored to the exact text	about 190
Quotation marks removed, as paraphrase	about 90
Invented sentences, removed	3

The invented sentences are the most instructive: on a page about Matt Haig, a final sentence that does not exist in the book had been glued onto a real highlight, while on a page about Alda Merini there was a line that seemed hers, “mette i primi al posto degli ultimi” (she puts the first in the place of the last), a Gospel reversal perfectly plausible for Merini, which however she never wrote.

The most common errors were more trivial, such as a changed word, a passage skipped without an ellipsis, a sentence from one chapter attributed to another, or an English translation passed off as the French original, and none of them was serious on its own, but together they affected about ten per cent of the quotations checked. Today the script runs at every update of the wiki, in a couple of seconds, and a page with a quotation that does not check out is not saved.

What I learned

1. The model writes well even when it is wrong, and that is exactly the problem, because a well-written error goes unnoticed and the quality of the prose says nothing about faithfulness to the facts.
2. The model must not check itself, because the check done by rereading the pages said everything was fine, while the scripts found the broken links, the lost highlights and the wrong quotations; the model is very good at writing those scripts, but much less reliable when it has to do the script’s work by eye.
3. The most dangerous errors are the silent ones, because a program that crashes gets noticed immediately, while one that skips a quarter of the data without saying so is discovered only by chance.
4. Every page must say where every sentence comes from, and the rule that now holds in the wiki is simple: quotation marks mean the exact text, taken from the highlight or from the book, and everything else is paraphrase and is written without quotation marks.
5. Separating who decides from who executes helps, and the most important corrections came precisely when, instead of trusting the agent’s report, I asked another model to go and look at the files.

And now?

Today the wiki is smaller than it first seemed, in the best sense of the word, because it says less, but what it says can be verified. After the quotations I also had the facts checked, such as prizes, dates, the structure of the books and who speaks in which chapter, and for this a script is not enough, rereading is needed: one agent per book compared the 46 longest pages with the full text and found, out of about 3,100 statements, one in six to be wrong, and not in the details, given that in a book by Michael Lopp three chapters were described backwards and in another two definitions turned out to be inverted.

I then moved on to the thematic pages, the ones that bring many books together and that are worth the most to me, asking whether the cited book really said what the page made it say, and only two statements out of three held up fully, with the thesis of a physicist and theologian reported upside down and the ending of a novel described as the opposite of what it is. The rule that emerges is that the further the text moves from the source, the more it gets wrong: the quotations were right nine times out of ten, the facts about the books five times out of six, the syntheses two times out of three.

In the next post I will talk about the other wiki, the work one, with scientific papers instead of novels and CERN’s Future Circular Collider, and with problems of a very different kind, such as what happens when a model reads the LaTeX source of a paper, including the comments the authors never meant to publish.

(This is the English translation of this post)