Is Amazon is training its AI on private health records?
I've been a customer of One Medical for many years. I was devastated when Amazon bought them in 2023. Despite my chagrin at this acquisition, quality of care did not change when Amazon took over, so I have stayed using the service.1
Health AI
Earlier this year, Amazon began to force all One Medical customers onto its new "Health AI" contraption. Amazon Health AI is a privacy and hallucination nightmare.2
It obviously uses patient health records in at least its context window, since it can theoretically answer questions about appointments and health ailments. But I was curious if private health data is also used in Amazon AI's training itself.
Spurred by this curiosity, I asked Amazon to restrict my protected health information (PHI) from any potential training, and to request a list of past disclosures of PHI.3 Privacy requests are met with an auto-reply saying I would get a response in 3 business days.
Instead of 3 business days, it took over two months and this is the answer I received:
Amazon’s response (emphasis mine)
We are denying your restriction request. We want to assure you that we take the responsibility of safeguarding and protecting your health information seriously.
At Amazon Health Services we use customers’ protected health information (PHI) to make getting care easier, and to ensure patient safety. We use customer data responsibly to improve experiences.
With privacy at the forefront, we’re building health care experiences that are more convenient and usable, while also protecting customers’ and patients’ privacy.
I requested additional clarification, asking if they would disclose how PHI may be used in AI training. I did not receive any additional response, this was in late March.
Hard to draw conclusions based on this. But if you're Amazon, and you've decided all of this is above-board and HIPAA-compliant, why wouldn't you "anonymize" this type of data and feed it into your training sets?
Can you just opt out of the entire feature?
No.
If you ask One Medical support about opting out of Health AI, you will be given a boilerplate response saying "Your conversations with the Health AI assistant in the One Medical app are protected with the same stringent, HIPAA-compliant privacy and security safeguards that have always protected your One Medical records."
I have asked so many times to opt-out that they now auto-close my tickets without even replying.
The P doesn't stand for privacy
HIPAA has some marginal protections for individually identifiable health information, but it's unclear how any of that is relevant here. If Amazon were training on health data it would likely be non-identifiable. I don't doubt that Amazon is complying with HIPAA, I'm just not sure that means much.
Have thoughts? Email me!
After Amazon initially acquired One Medical, I did leave. But I missed my great primary care doctor, so I came back. Funnily, that primary care doctor and virtually every other One Medical physician I've talked to has also been openly hostile towards this takeover.↩
Fun things Amazon Health AI has hallucinated about me and my health: injuries I've never had, things that were never said in appointments (if I go to the Health AI tab right now it tells me in my latest appointment my doctor took me off naproxen... something he did not do and a medication I am not on), and appointments I have not scheduled.↩
Amazon did allow me to get a report of past disclosure of PHI.↩