xssfox

I usually don’t write blog posts on things I’ve been doing at work, however AWS Backup is just so truly bad I want to warn people of some of the gotchas should you want to perform your own migration to it. Lets start with what AWS Backup thinks it is. (Emphasis by me)

AWS Backup is a fully managed service that centralizes and automates data protection across AWS services and hybrid workloads. It provides core data protection features, ransomware recovery capabilities, and compliance insights and analytics for data protection policies and operations. AWS Backup offers a cost-effective, policy-based service with features that simplify data protection at exabyte scale across your AWS estate.

Now before I tear AWS Backup apart let me start with the parts that I think are good and why you’ll probably use it.

• Automated and fairly straight forward copy jobs to other AWS regions and accounts
• Very easy to demonstrate to auditors that your backups are happening and working
• Can backup resources based on filters - no need to create a job for each resource
• Various methods of protecting the backups from tampering
• Continuous backup modes*

Cost effective

AWS product page mentions the cost-effectiveness of AWS Backup and this might be partly true depending on how you are currently doing your backups or what resources you are backing up. For example say you had a script that copied from one S3 bucket to another as your backup. In us-east-1 you would pay $0.023 per GB for that backup. Now that same backup using AWS Backup would cost just $0.05 per GB…. wait what. You are paying effectively 2.2 times more than S3 to backup using AWS Backup. What. DocumentDB, RedShift and Aurora are cheaper to backup using AWS Backup than S3. In Sydney region EBS volumes are cheaper to backup than S3. This makes no sense to me.

But wait, there’s more. When you perform the initial backup for S3 (and for changes) AWS Backup performs the requests to S3 like a normal client, this means you get stung with KMS, CloudTrail, S3 requests and GuardDuty api calls. There is no way to filter our AWS Backup from CloudTrail - so if you have have a lot of objects you could get up for thousands of dollars for that initial sync. AWS Backup team solution is to “turn of CloudTrail” or “don’t use AWS Backup”. Amazing. GuardDuty isn’t even listed in the support docs on possible costs.

Now before we leave the cost effectiveness section of this blog we should talk about budgeting. It is impossible to estimate how much AWS Backup will cost. I lodged 2 tickets to get an idea of how continuous backup mode works and how much it would cost to backup RDS and Aurora instances. The responses were both unhelpful and misleading. Copy jobs for Aurora and RDS are only snapshots, not PITR - this is not made clear in the docs. Trying to estimate this cost is near impossible because not even AWS knows how to do this.

Fully managed

There’s a bunch of fundamentals that are just missing from AWS Backup compared to pretty much any other backup solution. For example want to test that a backup plan works correctly? Guess what, you can’t trigger a manual start. You have to wait for the scheduled run time. So you want to get a list of jobs that are not successful - the UI only lets you filter by one state, and there is like 4 different failed states.

But it’s ok because you have useful AWS features like sending SNS notifications when backups fail… except for some reason you can’t send that notification failure SNS to another AWS account for some unknown reason….

Restore testing sounds great. You schedule it to test restores, you can even run your own validation scripts as Lambda functions. However it was clearly a hacked on feature. For S3 you have to restore the entire bucket. Have a large bucket, that’s gonna cost you. For Aurora and DocumentDB the restore test doesn’t even start an instance. It just creates a cluster. What’s being tested? Then to top it all off, S3 buckets linger around for days because to clean up the S3 bucket AWS Backup uses a lifecycle rule to delete all the objects (I know this is a good move for cost effectiveness but that’s an internal AWS thing!).

If you are configuring restore testing, much like backups there’s no way to trigger a test now. Hope your IAM, Lambda function, Event Bridge rules are perfect. Oh btw, since you have create resources for DocumentDB and Aurora - hope you can handle waiting longer than the Lambda timeout to do the test. Restore testing doesn’t even try to restore with the same configuration - so you have to manually define the VPC and SGs for the databases as well - otherwise it will try to use the default VPC and SG.

And how do you define those parameters for setting security group and vpc? As a JSON object? As a string? Nope, a JSON object as a string. There’s no validation on this, so if you send the wrong object, or send an array instead, the UI explodes.

It’s ok, restore testing will clean up your resources once the test is done…. nope if you created instances you need to delete them before it will clean up the cluster.

Simplify data protection

Here’s an incomplete list of gotchas for AWS Backup

• Remember to exclude the CloudTrail logging bucket so you don’t make loops
• Remember to exclude the server access logging bucket so you don’t make loops
• Don’t remove S3 Event Bridge notifications on buckets that have been configured with AWS Backup otherwise the backup has to start again (terraform does this by default when you have a notification policy configured)
• If the bucket is empty or not files that will be backed up - the job will be marked as failed
• Don’t configure AWS Backup for the same backup window as RDS backup window otherwise backups will branch creating conflicting backups
• Can’t create overlapping schedules for continuous backups - so be careful with your resource selection

Restore testing UI doesn’t display what resource it restored either. Just what the restore point ARN is. This makes it hard to demonstrate to an auditor that you tested resourcing of that resource.

Compliance insights

So after all of this you think, at least I have audit frameworks. Except if you are doing restore testing you quickly find out that the restore test resources are included in the report - suggesting you should backup your restores. And there isn’t a good way to exclude them because it’s really just AWS Config in a trench coat.

Project Horus is a high altitude balloon project. My first memory of Project Horus was when they launched Tux into near space at Linux Conf Au 2011 - my first Linux Conf Au. It would be awhile before I would start working on high altitude balloon related things. In that time Project Horus moved from using RTTY (radio teletype) for the payload telemetry to a 4FSK mode called horusbinary. Today horusbinary is used by many different amateur high altitude balloon payloads and supports additional custom fields.

Typically horusbinary payloads are decoded using the “horus-gui” app by connecting a radio to a laptop or computer via soundcard interface.

Mark’s done a great job packaging this up to make it easy for receiving stations to get up and running quickly.

Authors

Written by: Python Library - Mark Jessop vk5qi@rfhead.net

FSK Modem - David Rowe

FSK Modem Python Wrapper - XSSFox

And apparently I made the python to c wrapper using clib… I don’t even remember this.

For more casual users or people using phones and tablets could we build something more accessible? Recently I played around making a web frontend for freeselcall using web assembly which prompted Mark to ask the question:

but yeah, this is interesting, and suggests a horus decoder is probably possible

since i think you wrapped fsk_demod in a similar way for freeselcall as you did for horusdemodlib?

Introducing webhorus

webhorus is a browser based recreation of horus-gui allowing anyone to decode horusbinary balloon payloads from a browser providing a “no install” method and usable from mobile phones. It’s even possible to decode telemetry by holding up a mobile phone to the speaker of a radio.

webhorus is fully functional and is fairly close in feature parity to horus-gui.

How it was built

So horus-gui uses python library horusdemodlib which in turn uses the horus_api c library. That means if we build a website we’ll end up with the trifecta of Python, C and JavaScript in a single project.

horusdemodlib

I decided that I wanted to do minimal changes to the existing C and Python library and decided to include horusdemodlib as a git submodule. I had to request some changes via PRs to fix some bugs and make things easier to work with but it was fairly minimal.

One thing I didn’t want to use however was ctypes. The existing module uses ctypes and it has some drawbacks. The two biggest for me is that its very easy to get things wrong with ctypes leading to memory corruption and the other is that the current approach requires a dynamic linked library thats built beside the module, not with.

Since I’m packaging this for the web I decided to use cffi . cffi lets you define a c source file - usually you can just use your .h file¹ - that it uses to build a compiled python module with.

struct        horus *horus_open_advanced_sample_rate (int mode, int Rs, int tx_tone_spacing, int Fs);
void          horus_set_verbose(struct horus *hstates, int verbose);
int           horus_get_version              (void);
int           horus_get_mode                 (struct horus *hstates);
int           horus_get_Fs                   (struct horus *hstates);
....

horusdemodlib is simple enough that I could replicate the cmake process inside FFI().set_source()

ffibuilder.set_source("_horus_api_cffi",
"""
     #include "horus_api.h"   // the C header of the library
""",
      sources=[
        "./horusdemodlib/src/fsk.c",
        "./horusdemodlib/src/kiss_fft.c",
        "./horusdemodlib/src/kiss_fftr.c",
        "./horusdemodlib/src/mpdecode_core.c",
        "./horusdemodlib/src/H_256_768_22.c",
        "./horusdemodlib/src/H_128_384_23.c",
        "./horusdemodlib/src/golay23.c",
        "./horusdemodlib/src/phi0.c",
        "./horusdemodlib/src/horus_api.c",
        "./horusdemodlib/src/horus_l2.c",
      ],
       include_dirs = [ "./horusdemodlib/src"],
       extra_compile_args = ["-DHORUS_L2_RX","-DINTERLEAVER","-DSCRAMBLER","-DRUN_TIME_TABLES"]
     )   # library name, for the linker

Using this approach along with setuptools (for me via poetry) means that the c library is built as part of your normal pip install or package build.

I wrote a very small wrapper around the cffi library

import _horus_api_cffi

.....

self.hstates = horus_api.horus_open_advanced_sample_rate(
	mode, rate, tone_spacing, sample_rate
)
horus_api.horus_set_freq_est_limits(self.hstates,100,4000)
horus_api.horus_set_verbose(self.hstates, verbose)
.....

This is considerably easier than ctypes as you typically have to define the args and the return types. Any mistake will lead to at best a segfault and at worst undefined behaviour.

I utilised existing helper functions from horusdemodlib python library to ensure easy maintainability and decoder parity.

From here I can perform a pip install ./ and obtain a functional modem. So we’ve replicated what the horusdemodlib python library could do, using horusdemodlib and in Python…. wait what were trying to do again… right we needed static link python library - check.

from webhorus import demod
horus_demod = demod.Demod()
frame = horus_demod.demodulate(data)

pyodide

Next step is getting this from a compiled python library into web assemble to call from the browser javascript. Since we have python building the c library for us pyodide pretty much takes care of the entire process.

Once pyodide is installed and activated its literally just a case of doing:

pyodide build 

Everytime I’ve performed this I’ve found a handful of compiler issues to clean up. Nothing major. In this case it was just ensuring that one of the functions always had a return path. The output of this build process is a python whl file compiled as wasm32. I always think this step will be the hardest but its actually been the easiest so far.

Python in the browser

From the Javascript side with Pyodide installed² we can start accessing Python. I opt’d for putting the python code in its own file so that my IDE stopped getting confused. One important note here is that the pyodide library “.wasm” file needs to be served with mime type “application/wasm”

const pyodide = await loadPyodide();

// so we can actually load up micropip and install our packages and dependencies via pip - great for development however for production release I didn't want to rely on a third party server.
pyodide.loadPackage("./assets/crc-7.1.0-py3-none-any.whl")
...
pyodide.loadPackage("./assets/webhorus-0.1.0-cp312-cp312-pyodide_2024_0_wasm32.whl")
pyodide.runPython(await (await fetch("/py/main.py")).text());

What’s really neat is just how good the integration between python and Javascript is. Pyodide have done an amazing job with this. Using things like getElementById from within python just feels so wrong - but works so well.

horus_demod = demod.Demod(tone_spacing=int(document.getElementById("tone_spacing").value), sample_rate=sample_rate)

Adding audio

The last piece of this puzzle is audio. We need to capture microphone input to feed it into the modem. This was so challenging to get working. Jasper St. Pierre wrote a good blog post on “I don’t know who the Web Audio API is designed for” and I definitely feel that frustration on this project.

Now there’s some extra frustrations with what we are doing. We need as clean unprocessed audio as possible. I found out the hard way that Chrome tries to optimise audio for video calls. But before we get to that we need to get access to the audio device. This must happen through user interaction - so we force the user to hit a “Start” button, create an AudioContext and eventually perform a getUserMedia. On most browsers this will pop up a consent dialog requesting access to microphone. Prior to this we can’t even list the audio devices.

Now that we have consent and we’ve opened and audio device, then we list what audio devices the user has. To get the unprocessed audio we have to perform another getUserMedia with constraints set to remove any filters. The ones we care about are echoCancellation, autoGainControl and noiseSuppression. However if we blindly request these constraints turned off we’ll break Safari. So first we do a getCapabilities on the device first to see what we can set.

It’s all a bit of a hassle and getting the order of operations correct so that each browser is happy is a bit frustrating. Opening the microphone was only half the battle, we also need to obtain PCM samples to put into the modem. The only³ non deprecated way to do this seems to be via the AudioWorkletNode. This runs a bit of Javascript in a node / sandbox / thread. This actually sounds ideal - I could run the modem in its own thread and just send back decoded messages. However the sandbox is so limited that loading the modem into that environment is just too hard. Instead we just buffer up audio samples and when we have enough use this.port.postMessage to send buffers data back to the client. Oh and because this is the Web Audio Api we need to convert -1 to 1 floats to shorts. What fun.

Anyway, that’s all the tricky parts glued together. The other 80% is building out a UI with Bootstrap, Plotly, and Leaflet + figuring out how to get Vite to build it into a solution. The code is up on GitHub if you’d like to check it out, including a GitHub Actions workflow so you can see the entire build process.

(tl;dr - If you are using a cheap smart watch, probably don’t. If you want to skip to the juicy bit jump to “Authentication”)

As with most complicated things in my life, it started with a simple question.

Do you know how to get the data off this watch?

Surely there’s an export feature in the app to a common file format like GPX, TCX or FIT. That’s pretty standard. Yet we couldn’t find a way. The only share and export functionality just provided an image with some stats on it.

Also given Android’s lockdown on the file system (to the point that you can no longer backup your phone!) there was no way to access the DB to obtain data (without root anyway).

Investigating the Ryze Android app I quickly discovered that this is a white label of an IDO smart watch.

It got me thinking - could I pick up one of these watches for a lot cheaper to play with. Eventually I landed on searching Amazon. The hardest part was working out if it was actually an IDO based device. I discovered what worked well was looking for watches that use the “VeryFit” app which is the IDOs stock app. There’s actually a bunch of watches that use this app. I found one for a low price of $39 (AUD) and free next day shipping.

Hilariously I paired my IDO watch with the Ryze app and it just worked fine. I wasn’t actually expecting that to work but it seemed ok with it.

Would I buy one? yes - as a Bluetooth Low Energy reverse engineering example device otherwise no. Onwards to reverse engineering!

To the cloud

The first thing I looked at was syncing data from the cloud. This was pretty easy and basically just required following the same request flow as the phone. It however has some drawbacks, you still need to use the app, and you need to have an account. I didn’t really progress any further with this - but it is a viable way forward with those limitations.

Getting data off watch directly

I’m going to skip past dealing with authentication for the moment as that gets a little spicy and we don’t want to ruin the standard story telling plot framework.

Once connected and authenticated the device presents a bluetooth service at 0AF0 which provides several characteristics. The import ones to us are a subscription to 0AF7 - this is where data from the watch to the phone is sent while 0AF6 allows us to send commands.

adb logcat and BlueSee.app were lifesavers here. The app logs a lot of information about what its doing and how its working. You can pretty much work out everything from using the app with adb logcat running then emulating that flow with BlueSee.app. Also what helps is that the developers seemed to have a scattering of their documentation and source code on GitHub and some of the other vendors shipping these have done the same. Not all the source code is available which would have made reverse engineering just a little bit easier but its a good start.

So we send commands by writing to 0AF6. The easiest first set is 0x0201 which provides some basic device info. The device will write back to 0AF7 with the returned data. For these 0x02.... commands the return data will be structured like:

02 - Header
01 - CMD (get basic info)
.. - Reply data structure (device)
.. -                      (firmware version )
..
..

Data is little endian.

V3 protocol

Getting activity data we need to use the “v3” protocol. These commands have the first byte a 0x33 or “3” in ASCII.

The v3 protocol can have data be split over multiple transmissions - a requirement of using this style BLE. So that the watch and phone can know that a new command has started rather than a continuation there’s a preamble DAADDAAD. This is just an utterly weird design choice given all the other features in BLE that could have been used. There’s then a service selectorish thing, length, command, sequence fields, followed by data and eventually a CRC.

To send a request to get the last activity I use the following command:

0x33 - '3' version protocol
0xDA - this is preamble to detect new command midway through
0xAD - preamble
0xDA - preamble
0xAD - preamble
0x01 - health?
0x10 - length (without '3' and checksum (last two bytes) - aka minus 3 on single packets)
0x00 - length
0x04 - cmd
0x00
0x0B - sequence - can just pick anything
0x01 - sequence
0x00 - start / stop
0x04 - cmd
0x00
0x00
0x00 - checksum
0x00 - checksum

Checksum calculation

I think its just some sort of CRC16 however I’m not sure which because the device I have never checked it so I never bothered working it out.

V3 return data

The return contains much of the same header. You can use the sequence numbers to line things up with you have to handle queueing of data and multiple requests.

Activity data is nearly always split over multiple packets so just read in the subscription until the data received matches the provided length in the header or until you see another preamble.

Then its a matter of unpacking the data. Noting that you’ll have arrays of varying lengths.

You might be thinking that this protocol will be stripped down to save transfer speed, however the protocol sends stuff like pace in several different units.

Online activity downloader

I built a really rough and dirty website for talking to my device and getting the health data off it. You can play around with it at https://sprocketfox.io/ido/. It’s happy path’d for my $39 crappy watch so probably won’t work for most people. It generates a TCX file that can be used on Strava or Garmin.

Authentication

You can pull activity data, sleep data, heart rate data, menstrual data. You can perform firmware updates. I don’t have GPS in my version of the watch but I bet you can access that as well. Whatever you want you can get it - there’s no protection. I can only test with my watch, but I suspect this applies to most devices that have been white labeled.

The watch I bought off Amazon has 5000+ ratings and the VeryFit app has over 5 million downloads. There’s 10k Ryze Connect app downloads according to Google Play. From what I can tell Cove, BoAT and bfit Move are also using the same app/protocol - I imagine there is a lot more as well. These are sold in retail stores like JB Hifi and Harvey Norman. There’s even kids versions of these watches which adds to a creepyness factor.

I pondered about how to disclose this and decided to go with the “fuck it” approach. The reasoning is that it’s going to be hard to find all the different vendors to let them know and I suspect they won’t care. I just don’t have the time go talk to AI chat bots, lodge support forms, have support people not understand my concern, have multiple generic replies saying “it’s not an issue our vendor said its secure” ect…. Is “JJMJKJ” (yes thats who sells it on Amazon) going to do anything - no. Not a chance. So rather than go through that entire process, get burnt out, and still be left with no actual device security I’m taking the approach of letting people know as soon as possible.

If you’re in Australia you can request a refund as the device isn’t fit for purpose or of acceptable quality.

Git

Details about this can be found at https://github.com/xssfox/idowatch

Unexplored security fun

• File transfer / bluetooth serial endpoints: There’s some file upload for firmware features that could be interesting
• Hijacking the app by pretending to be a watch: Perform attacks the reverse way, pretend to be someone else’s watch. Maybe can make phone calls, take pictures.
• Overflows: There’s lots of places that smell like overflows can happen.