I Built a Local-First HSA Receipt Tracker with Flask, Google Drive, and AI

A tiny local python app that turns HSA receipts into structured records in Google Drive and Sheets in about five seconds.

Last year my family switched to a High Deductible Health Plan (HDHP) with a Health Savings Account (HSA).

HSAs are unusual in the U.S. tax system because they’re triple tax advantaged:

• contributions are tax-deductible

• investments grow tax-free

• withdrawals for qualified medical expenses are tax-free

The Bogleheads wiki has a great explanation if you're curious:

https://www.bogleheads.org/wiki/Health\_savings\_account

Because of those advantages, some people treat their HSA as a long-term investment account instead of reimbursing medical expenses right away. Almost everyone I tell about this has a 🤯 moment because they’ve never heard of it. So, if you're one of those lucky people to learn about it today, you're welcome!

Your wallet loves this one weird trick!

The strategy looks like this:

1. Contribute to the HSA

2. Invest the money

3. Pay medical expenses out-of-pocket

4. Save the receipts

5. Reimburse yourself years (or decades) later, or don't

6. Profit

There’s no deadline to reimburse yourself as long as:

• the expense happened after the HSA was opened

• you kept documentation

That last part is the problem.

Here’s what the result looks like after uploading a receipt:

Every receipt automatically becomes a structured record. Each row links back to the original receipt stored in Google Drive.

Date	Vendor	Amount	Receipt
2026-02-14	Quest Diagnostics	$87.43	Drive Link
2026-02-03	Walgreens	$14.29	Drive Link
2026-01-19	Dentist	$120.00	Drive Link

---

The Receipt Problem

If you're saving receipts for potential reimbursement later, you need to keep track of:

• provider

• date

• amount

• proof of payment

• the original receipt

Manually this usually turns into:

• PDFs scattered across downloads folders

• random email attachments

• a spreadsheet you forget to update

I was doing this manually with Google Drive and a spreadsheet, and it wasn't too hard, but... I wanted something simpler.

Drop in a receipt → have everything filed automatically.

---

The Idea

I built a tiny local-first web app that:

1. accepts a medical receipt PDF

2. extracts useful fields automatically

3. stores the document in Google Drive

4. logs the expense in Google Sheets

The whole flow takes about 5 seconds of actual work.

---

Demo

Here’s the entire workflow in real time (~5 seconds of actual work - the rest is me moving slowly for the video's sake):

{% embed https://vimeo.com/1171880979?share=copy&fl=sv&fe=ci %}

Upload a receipt → parsed → stored → logged.

---

The Stack

The app is intentionally small:

• Flask – local web app

• pdfplumber – extract text from receipts

• OpenAI (optional) – prefill receipt fields

• Google Drive API – store receipts

• Google Sheets API – expense log

The workflow looks like this:

receipt → upload → text extraction → AI field parsing → Drive storage → Sheets entry

---

What Happens When You Upload a Receipt

When I submit a receipt, the app:

1. saves the uploaded PDF locally

2. computes a SHA-256 hash to detect duplicates

3. extracts text using pdfplumber

4. optionally asks the model to extract fields like:

• vendor

• service date

• amount

• payment date

• payment method

• notes

1. creates (or reuses) a Google Drive folder for the month:

   2026-03/

2. uploads the receipt there

3. appends a row to Google Sheets with a link to the file

Example spreadsheet row:

Date	Vendor	Amount	Receipt
2026-02-14	Quest Diagnostics	$87.43	Drive Link

Now every expense has:

• structured data

• the original document

• a searchable log

---

One Small Feature That Ended Up Being Useful

Duplicate detection.

Each uploaded file is hashed and stored in a local receipt_hashes.json.

If I accidentally upload the same receipt twice, the app can catch it before cluttering the spreadsheet or Drive folder.

Tiny detail, but it prevents a lot of mess.

---

Why Local-First?

This could easily be a SaaS product.

But for personal admin tools, local-first is usually better:

• no extra account to maintain

• easier to hack on

• my files stay in my own Google account

• no database or infrastructure to run

I also set it up as a macOS launchd service, so it’s just always available on my laptop.

The goal wasn’t to build some big product. It was to remove just enough friction that I’d actually keep my receipts organized.

Now the workflow is:

Download receipt ↓ Upload ↓ Done

Which is about the amount of effort I’m realistically willing to spend on personal finance paperwork.

---

Repo

If you're interested:

https://github.com/pjhoberman/hsa-tracker

---

If you’re using the "save receipts and reimburse later" HSA strategy and built something similar, I’d love to hear about it.

Or if you have a better way to automate this - please tell me before I add OCR and accidentally build an entire product.