A Profile Can Be Accurate and Still Be Wrong

Three late assignments can be three correct entries in a student record. They can also become the sentence “poor engagement.” The facts did not disappear; context did. AI can make that distance easier to cross—and harder to notice.

The student hidden inside the summary

A student submits three assignments late and misses two morning seminars. Each entry in the university system is correct.

An AI assistant organises the record for an academic adviser and produces a concise description: “declining engagement and inconsistent participation.” The sentence is fluent, useful, and connected to real events. It may also be wrong in the way that matters.

The student works late shifts and has recently started taking a younger sibling to school. They remain interested in the course, watch recordings after work, and have not known how to explain the change. None of that context appears in the attendance and submission fields.

If the summary prompts an open conversation, it may help the adviser notice a student who needs flexibility. If it becomes a warning category, shapes a reference, or travels into another decision, the same sentence becomes an institutional representation with consequences.

The danger is not only that AI may invent a fact. It is that AI may arrange accurate fragments into a coherent story, and coherence may be mistaken for understanding.

AI makes this easier to scale because it can work across information that institutions previously kept apart. Messages reveal relationships. Calendars reveal routines. Documents reveal health, work, education, migration, finance, and family. Combining fragments can recover overlooked evidence. It can also produce a claim that existed in none of the sources.

The information was present. The representation was produced.

Carry this forward

A profile becomes dangerous when its neatness hides how much of the person remains unknown.

Why institutions need representations

An institution cannot treat every decision as if it knows nothing about the past. A doctor needs a clinical history. A university needs an academic record. A welfare office needs evidence of circumstances. A lender needs some account of capacity and risk. Representations help organisations remember, compare, coordinate, explain, and act consistently.

AI can make that work faster. It can locate overlooked evidence, translate documents, identify contradictions, and reveal where a record is incomplete. For a person confronting a complex bureaucracy, a well-designed system may make the institution more understandable rather than less.

The alternative to a profile is not always a complete human encounter. It may be a hurried worker, an inaccessible archive, an arbitrary impression, or no decision at all.

The problem is therefore not representation itself. It is forgetting that every representation selects, simplifies, and serves a purpose.

What AI changes

AI changes the reach of representation.

It can combine more sources, infer unstated relationships, generate fluent explanations, update a profile continuously, and reproduce the result across several systems. A tentative interpretation can travel without the uncertainty that originally surrounded it.

This creates a political economy of knowledge. Power belongs not only to whoever owns the data, but to whoever can turn fragments into a credible account, place that account inside a workflow, and make others treat it as true.

The resulting profile may be difficult to challenge precisely because no individual authored it. One provider supplied the data, another generated the inference, an institution stored the summary, and a worker relied on the category.

When knowledge production is distributed, responsibility can disappear between its contributors.

From information to institutional judgment

Institutions have always acted through representations: files, applications, references, scores, categories, assessments, and professional judgment. AI does not invent this practice. It changes its speed, scale, opacity, and reach.

A useful analysis follows a representation through separate questions:

1. Access: which information and operations were available, about whom, and for which purpose?
2. Representation: what summary, category, prediction, ranking, or generated account was produced?
3. Provenance: which parts were provided, observed, linked, derived, inferred, generated, or unknown?
4. Persistence: where was the representation stored, copied, transferred, or combined?
5. Availability: which people and systems could retrieve or use it?
6. Reliance: did it become an operative premise of institutional judgment?
7. Consequence: whose treatment, options, record, or burden changed?

The arrows matter. Access is not power. A generated statement is not knowledge. Availability is not reliance. A decision after a profile does not prove that the profile caused it.

Where evidence is missing, the honest answer may be: we do not know.

Accuracy is not justification

Suppose a system correctly records that a person changed address three times in two years. An institution may infer instability. But the person may have moved for education, seasonal work, family care, conflict, or immigration status.

The addresses can be accurate. The institutional account can still be unjustified.

Justification asks more than whether a fact is correct. It asks whether the source is adequate, the inference warranted, the context relevant, the uncertainty preserved, the purpose legitimate, and the use proportionate.

This matters especially for migration. People whose work, families, documents, and entitlements cross jurisdictions are often represented through records produced for different purposes and legal systems. A gap may mean fraud, or it may mean that one institution cannot see another institution’s document. A mismatch may mean deception, or translation, delay, or a change in legal status.

Fluency can make a weak inference look complete. A polished paragraph is not evidence of understanding.

Correction is not repair

Imagine that a person successfully corrects an address or explains a bank transfer. Is the problem over?

Not if the earlier information has already been transformed. A risk marker may remain inside a profile. A generated summary may have copied the marker. A worker may have pasted the summary into an official record. Another service may have received the old account.

Source correction changes the beginning of the chain. Repair must reach its consequences.

That may require revising the profile, locating dependent records, notifying recipients, reconsidering decisions, restoring an opportunity, compensating an irreversible loss, and checking whether the same failure affected others.

Deletion is not always the answer. A restricted record may be needed as evidence of mistreatment. The question is whether the disputed representation continues to govern the person.

Explanation is not contestability

An institution can explain how a score was produced while leaving the affected person unable to change anything.

Contestability requires a route to introduce counterevidence, identify what was inferred rather than observed, preserve uncertainty, suspend harmful reliance, obtain competent reconsideration, and communicate correction to dependent systems.

This does not mean that people know themselves perfectly or may veto every institutional claim. People can be mistaken. Institutions may hold legitimate evidence and competing duties. Epistemic agency means that a person can participate in the process through which consequential knowledge claims are formed and resolved.

A decorative explanation attached to an irreversible outcome is not meaningful contestability.

Govern the representation, not only the data

Protecting source data is necessary, but it does not govern everything that can be produced from those data.

A defensible arrangement would distinguish access to information from authority to create a persistent institutional account. It would make permissions purpose-bound and temporary. It would label whether a claim was provided, observed, linked, inferred, or generated. It would preserve material uncertainty and prevent a low-confidence account from silently becoming a categorical record.

It would also restrict propagation. A representation created to answer a private scheduling question should not automatically become available to an employer, insurer, lender, university, or public agency.

Where an AI system materially contributes to a consequential judgment, the institution—not the model—must own provenance, justification, contestability, and repair.

Counterarguments and limits

Complete lineage is technically and administratively expensive. The objection is valid. A private, low-consequence draft should not carry the same burden as a representation used to deny an essential service. Requirements should scale with institutional reliance and consequence.

Users cannot inspect every internal representation. Nor should they necessarily receive trade secrets, security-sensitive details, or other people’s data. Meaningful access can take the form of reasons tied to the actual institutional basis, material information, independent review, and a route capable of changing the result.

Alternative interpretations may improve contestability. A second analysis can reveal uncertainty or expose a weak inference. But multiplying interpretations is not enough unless the institution can reconsider the representation it actually relied upon.

Institutions need to infer. Medicine, insurance, education, welfare, and public administration cannot operate only on facts supplied by the affected person. The framework does not prohibit inference. It requires institutions to distinguish inference from knowledge and justify consequential use.

The framework cannot resolve conflicting rights. Privacy, access, due process, safety, equality, trade secrecy, expression, and public interest can conflict. Jurisdiction-specific law and legitimate institutional judgment remain necessary.

Conclusion

Institutions need representations, and AI can help them find evidence, coordinate work, and make complex processes more accessible. But a representation can acquire authority merely by appearing coherent, complete, and available at the right moment.

An AI system can create institutional knowledge claims from information a person never intended to become a profile. Governance must therefore follow the representation: where it came from, how it changed, where it travelled, who relied upon it, and whether a successful challenge can repair what followed.

Information can describe a person. Institutions decide when that description is allowed to stand in for them.