Transactions are grouped into blocks, each block references the one before it cryptographically, and the whole chain is stored by thousands of independent nodes. Changing an old entry would require redoing every block after it on a majority of those machines.
What it removes is the trusted intermediary. Participants who do not trust each other and share no central record keeper can still agree on one ordered history, which is the actual invention.
The costs are real. Every node stores and verifies everything, so throughput is low and expensive compared with a database, and the immutability that protects the ledger also means an error or a theft cannot be reversed by anyone.