Sondehub

Strap in, this story involves a cheese fortune teller, the department of war, and nearly every other government department in between.

In 2017 (I think?) I was introduced to weather balloon hunting by Mark VK5QI. At the time the Australian balloon chasing community was small. Only Melbourne and Adelaide radiosondes (the transmitter on weather balloons) were being tracked on a website called Habhub - high altitude ballooning hub. This site was designed for amateur balloons and not meteorological weather balloons.

Over time more and more radiosondes were tracked on Habhub and eventually Habhub admins introduced a default filter that removed weather balloons by default. A query parameter could be added to the URL to remove the filter and on 12th of May 2018 sondehub.org registered with a single purpose - a URL redirect to Habhub with a radiosonde specific filter. To be clear - this was more of a joke than a decision to run a radiosonde tracking service. You’d go to sondehub.org and it would redirect you to habhub.org. That was it.

However Habhub was never designed for so many unique balloons each day. By July we decided to start proxying radiosonde ingestion data through SondeHub. This allowed us to capture more data as well (no longer rate limiting our selves). This went to a seperate OpenSearch cluster, however at this stage we didn’t use or expose this data. I was using this more as a toy - to play around with different Amazon Web Services (AWS) services and analytics platforms.

By 2019 the Habhub servers were really struggling - aprs.fi as well. We realised that we needed to run our own service and our initial plan was that we would build new APIs, and eventually new frontend. We then started getting information requests from government agencies regarding radiosonde data. For example we received a request regarding an insurance claim about a radiosondes hitting a horse, causing it to bolt through a fence. One of the reasons for this is because unlike official software at the time, our system tracked the radiosondes all the way to the ground.

Also in 2019 we detected a drop in radiosonde launches. This lined up with the GPS rollover date - we thought our software was broken however it turned out to be issues with Vaisala’s equipment which prevented launches from occuring. Funnily enough our software handled the rollover ok.

In 2020/2021 we ended up doing was building backwards compatible APIs for the Habhub frontend and started testing the Habhub frontend pointed at our backend. It mostly worked. We started receiving all the data rather than just partial data and providing open access to our data via S3. We even started running our own predictor - which is used by my entities today.

With our own predictor running Mark developed a system we call reverse predictions. This is where we take data from an already launched radiosonde and use the wind model to run the predictor backwards which determines a rough the launch location prediction. It works extremely well. We could detect a bunch of radiosonde launch sites that were poorly otherwise documented along with start assigning balloons to launch sites.

Our first taste of dealing with the military

Then in 2021 we received an email

sensitive/military/… installation. As such, we really prefer that it is not explicitly marked on any map.

The thing is though that wind data isn’t just used for predicting the weather. It’s also used to calculate artillery ranging. What we had started doing is accidentally mapping out artillery sites. We decided to keep reverse predictions but we delete launch sites on genuine requests.

The reverse prediction system has also detected many number of military vessels in the ocean.

Lots more development happened on SondeHub with features like websockets and MQTT for live feeds. We disconnected Habhub backend from our proxy and with grant funding from ARDC we were able to setup a prototype amateur high altitude balloon version of SondeHub.

Eventually Habhub was shutdown due to a lack of maintenance and we rushed together to migrate what we could to SondeHub.

$439,000 missile vs party balloon

All was going fine until the 2023 “China spy balloon” incident. SondeHub had a lot more traffic - but our architecture made it fairly manageable.

Then Feb 11th 2023 the US allegedly used AIM-9X Sidewinder to shot down an amateur radio balloon. That morning I woke to high usage alarms in my inbox. SondeHub had been linked to on the Washington Post. Our site managed to handled the extra traffic reasonably well.

Since then we’ve many support requests from .mil and .gov addresses. We’ve also had requests from aviation industry / air control towers.

In Dec 2024 - alarms in my inbox again. This time getting alarms for predictions. Someone decided to smash our api. This seemingly starts happening every week.

Full scale invasion

We turn on logging. The requests coming from a single IP. We had some suspicions that a private company was using our backend to generate predictions. We poke their website to see - sure enough they are - an angry email to them. However they weren’t the problem.

We ask some people.

Note that the precision of these points has been intentionally been reduced. This data is also significantly old and does not show the entire dataset. This blog post has been delayed until balloon warfare was more common knowledge.

Fuck. And Fuck Russia.

(for time travellers and people in the future - in 2022 started a “special military operation” - aka a full scale invasion into Ukraine. The war continues at time of writing. Fuck Russia)

Suddenly my mind was filled with ethic and legal questions. We also suspected they aren’t using the API correctly. However we didn’t know how to get in contact.

We did eventually got some messages out via a contact

“We work with mHAB’s as you know, but some other groups likely fly fixed-wing and use Sondehub to help them “surf” the sky to target areas.

“Sent this in Ukrainian to a few milchats and will see what turns up: “I wish everyone good health. If anyone knows of a deep strike team that uses a python script with some open source wind forecasting engine, please contact me directly. They are causing numerous problems with queries, which can lead to them being blocked and they need to take action to be able to continue using the prediction system.””

I also quickly rushed together a docker compose file so anyone could quickly run their own predictor that wasn’t reliant on us.

Meanwhile (and you might have noticed me asking for AWS help on fedi) we contacted AWS as the source IP was from an AWS network. It was very important however to make sure the AWS support did not shutdown access.

Our messaging included:

It is incredibly important that the http request data is not distributed. It is also important that the source AWS account is not blocked, rate limited or terminated - loss of life could occur.

Something that I thought I’d never have to write in support emails. The messaging was important because I did not want the service cut off, and I did not want the data to reveal launch sites.

After a bit of waiting we received:

AWS reached out to me that a lambda function of mine was flagged for potentially scraping api.v2.sondehub.org and they told me to reach out to you to get this resolved.

We emailed back and forth and provided documentation on how to run the predictor locally.

Office of the Secretary of War (Intelligence and Security)

In 2025 we received a request for data from the “Office of the Secretary of War (Intelligence and Security)” (US). Generally if there’s mutual community benefit we’ll find, process and release the data for free. However given this is was the Department of War and no expected community benefit we decided they should pay for the data. I was hesitant even working with them, as I don’t really want to help military, let alone the US - but since our data is public if we didn’t do it someone else probably would. So my reasoning shifted to, may as well extract some funds to pay for SondeHub infrastructure at the very least.

An invoice was created and sent through - but never paid or followed up on. I have no idea why they were requesting the data or what it was about.

Other tidbits along the way

It hasn’t just been the military that we get emails from. Occasionally citizens who find radiosondes end up contacting us (often we don’t know how they even find us), along with a range of other organisations.

National Transportation Safety Board (US)

In September 2025 the NTSB contacted us. My first reaction was to search for news stories.

do you have information on any balloons in the Utah are between 1200 and 1300 UTC on 10/16/2025

We provided our data but also started hearing some rumours about a possible plane / weather balloon collision that was reported via ACARS. While none of the balloons tracked by SondeHub lined up, we did forward some information that a Windborne balloon was in the area. Windborne later confirmed this was the likely collision and have made several changes to their system to prevent future issues.

We have a number for you to call when you’re ready to copy

Please contact us as soon as practicable with more information… Contact our Operations Manager at

This was a really strange interaction for us. A tower(?) operations supervisor was requesting information about balloons in the area. The balloons in question were meteorological weather balloons. Not launched by amateurs. We had to explain that they are normally scheduled, not controlled, and fall (probably, not a lawyer) within Part 101.D of FAA regs. Along with that we didn’t have contacts or registration details of these launches.

We have a lot of Aircraft in the sky that don’t want to get too close to one of these balloons! Is there any way to coordinate more directly with the controlling entity, or to have them give us mission details and contact information ahead of time? It sounds like you guys have a big operation, I don’t know if this is a one off event or if you have systems in place to communicate these things

Explaining to the FAA that weather balloons exist wasn’t on my bingo card.

Hit and run

On 2/5 around 8pm was there a balloon located in Anamosa Iowa?

Someone recovered a radiosonde from a property but ran into a building along the way. They left without leaving a note. The property owner contacted us for help to locate the person.

Jam, tasty tasty jam

There’s a great site that uses ADSB data to track GPS jamming called gpsjam.org. We’ve also been detecting not only a lot of GPS jamming but also GPS spoofing. I always find the patterns interesting. I’ve been presuming that the pattern is either for making the impacted targets easier to identify or to crash the vehicle in a specific way?

The cheese fortune teller and other job titles we’ve seen over the years

Probably the most interesting job title we’ve had the pleasure of reading in an email is from Jennifer Billock, Freelance Writer and Author, Certified Tea Specialist, Cheese Fortune Teller. Jennifer wrote an article for STNDRDS about weather balloons which is outside our usual places of exposure.

During this time we’ve seen many job titles and subjects, I’ve started collecting them.

• [SEC=🌶️🌶️🌶️]
• Naval Air Warfare Center – Aircraft Division Maritime Patrol and Reconnaissance Aircraft (MPRA) Program
• Acquisition Program Manager Integrated Processes Branch HQ AFRL/XPOP
• Upper Air Quality Assurance Meteorologist Observing Systems & Operations, Data & Digital Group
• Senior Advisor for Safety and Quality
• Meteorologist Weather Forecast Office
• Manager Upper Air Network
• General Manager Observing Systems and Operations and Chief Engineer
• Meteorologist National Weather Service
• Field Research Manager, Center for Western Weather and Water Extremes
• Video Journalist, Visual Investigations - NY Times
• Senior Meteorologist, National Transportation Safety Board
• Operations Supervisor U.S. Department of Transportation/FAA
• STNDRDS Freelance Writer and Author, Certified Tea Specialist, Cheese Fortune Teller
• Meteorológiai fejlesztő (met.hu)
• SUNY Oswego Lab Technician Atmospheric and Geological Science
• SpaceBalloon Project
• Any many more

The weird

Most organisation and vendors are willing to work with us. This is because chasing radiosondes removes them from the environment and promotes citizen science. I asked “Meteolabor AG” for one of their radiosondes so that we could confirm compatibility. This is what I received back.

Official response from Meteolabor AG: For strategic reasons, we do not provide any data or sample devices. Our transmitters shut down after a certain period of time, at the latest when the battery capacity is exhausted. This is due, among other things, to strategic considerations.

We are aware of the so-called waste problem.

---

Personal comment: I would personally like to draw attention to military activities, particularly in the Middle East, which result in significantly (exponentially) more waste and toxic substances being released into the atmosphere and left lying around in the environment – or entering the food and water cycles

In addition to military operations, countless “missions” are currently being flown over Europe with the aim of leaving “contrails” in the sky [rather “chemtrails”]. I know their purpose; I know what NetZero is supposed to achieve, and what decarbonization and CO2 reduction are intended to accomplish. I am well-informed about the climate hoax.

Start there! The people to talk to are politicians, NGOs, and very wealthy old white men.

Which is… certainly something.

Onwards

I hope you liked this selection of SondeHub chaos. I haven’t included every interaction we’ve received over the years, so there might be a part 2 to this post in the future.

For those not familiar, Horus Binary is a high altitude balloon telemetry system. It’s goals is to overcome some of the challenges compared to other general purpose systems as high altitude balloons have unique challenges.

Challenges with high altitude balloon telemetry

Telemetry is important to track the location of the balloon to aid in recovery, but also to downlink data in case recovery isn’t possible. Often several payloads are flown and some flights contain multiple communication methods, not just Horus Binary. As with the majority of things mentioned in this post, the RF downlink is a series of compromises specifically chosen to solve a problem. For example you can run an entire DVB-S transmitter and have a live feed of the video footage - but for this you need yagis and a fixed location - limiting the possibility of recovery without another solution.

It’s pretty common for a high altitude balloon to reach 30km altitude. So even if you are standing directly under the balloon at that time, it’s some distance for wireless communication. While the distance is pretty great, the advantage of being a balloon is there’s rarely any obstruction between the receiver and transmitter - so we win back a little bit of SNR from having the line of sight advantage. To handle the distance we want something with a low baud rate and error correction. While typically we have line of sight, we still want to receive telemetry at launch and landing, where conditions might not be perfect for RF.

Balloons can only lift so much. Large balloons require more helium, more expensive, harder to launch and may require more regulation/rules. So the payload also needs to be light. Launching a IC-7100 radio isn’t a great option. But down sizing also imposes some more challenges. Smaller payload means less battery, less transmit power. Horus Binary is often transmitted from repurposed radiosondes, powered by one or two AA batteries with an RF output power in the tens of mW. The lighter transmitter solution allows for more weight dedicated to science.

Now one of our key requirements is trying to recover payloads. This means we want to receive a telemetry packet on, or as close to, where the payload hits the ground. We could build a system that let you send kilobytes of data - but at the low baud rate, it would take forever to send. These balloons are falling at rates around 20km/hr, so if we receive a packet every minute then very likely we might not receive a low enough altitude packet to accurately determine the landing location and recover the payload. So we want to keep the payload size short.

So what about existing infrastructure, like satellite, cellular , LoRa networks. These actually do get used on flights, but they have their shortcomings. For cellular there might not be coverage at the landing location, and since cell towers generally point their antennas down or to the horizon the coverage can be patchy in the sky. For satellite there’s weight constraints for high speed links, and for most small trackers the update rates can be minutes or more. LoRa is a similar story with low update rate (if being a good citizen) and you need to rely on good area coverage. So if we aren’t using existing infrastructure, then we need to bring our own. That means having recovery vehicles able to easily receive it, and a network of stationary receivers where possible.

For a mobile receiver this means we don’t want a fancy multi element yagi tracking system. We want a simple dipole or cross dipole that can used on a moving vehicle. Likewise on the payload, we want an antenna with little gain in any one direction, as we won’t be sure which way the antenna is facing.

Summary of requirements

• Low SNR requirement
• High update rate
• Reliable data
• Low power usage
• Mobile receiver

This results in some of our design constraints for a protocol:

• Low bit rate (100 baud)
• Small packet sizes (< 128bytes)
• Low power transmitter

Horus Binary v1/v2

Horus Binary v1/2 have been around for awhile now and pretty well established in the amateur high altitude ballooning space. It uses a well tested 4FSK 100 baud modem with Golay for error correction. The payload itself is 20 (v1) or 30 (v2) bytes, with an additional 2 bytes for a checksum.

Anyone can receive the telemetry by using either a sideband radio receiver or software defined radios like the RTL-SDR, which keeps hardware costs down.

We don’t need to dive too deep into RF for this post as this modem is well established and pretty good quality, but we do need some quick fundamentals. Horus Binary uses 4 times frequency shift keying (4FSK) - or put another way, 4 tones to indicate ones and zeros.

If you tune a radio to a frequency where there’s no station, you hear noise. This is also true for data modems. Without any sort of checking we’d get a random string of ones and zeros. (sometimes this is used to generate random numbers!). In fact without anything transmitting the modem doesn’t even know where the signal/tones are.

This is why at the start of an RF packet, often we transmit what’s called a preamble. A sequence that is super easy for the modem to figure out where the tones are. With the modem synchronized, the next problem is decoding. The modem itself doesn’t really know whats valid ones and zeros and whats invalid.

If we take the ones and zeros and try to decode each one and check its checksum, then we would waste a lot of CPU and possibly not even keep up with the incoming data. We really need a way of quickly seeing if a packet is actually likely to be a packet.

For this we use a “unique word”¹. A series of bits that are always at the start of a packet. We can accept a few of these bits to be wrong due to noise by setting a threshold of valid bits. Effectively don’t try to decode the packet unless the unique word is at the start.

Putting this all together for Horus Binary v1/2 we have

<preamble> <unique word> <payload> <checksum>
0x1B1B1B1B    0x2424      DATA(30) CHECKSUM(2)

Horus Binary v2 payload data

30 bytes is not a lot. So what is in there, and how do encode/decode it? For v1/v2 a simple struct packing is used. The table below shows how the fields are stored. By packing the data down as binary data without any tagging or delimiters a lot of useful information can fit inside a small packet.

The astute among you would have noticed the “Custom data” field. If there are no delimiters or field separators in the format, how does one decode that data into fields again. Like wise the Payload ID seems to be a number, rather than a callsign - but on sites like SondeHub a callsign is displayed.

Horus Binary v1/2 rely on two files that are regularly updated to resolve the payload ids to callsigns and the rules to unpack the custom data. This means that receiving stations need to have internet access prior to the flight to get the latest data and launch operators need to submit pull requests to get their callsign and custom data allocated.

The smallest a custom field could be was a byte.

Handling different sized payloads

Horus Binary v1 and v2 use different payload lengths, however receivers don’t need to configure which version they are receiving. How does that work? We try both

In v1/v2, we have a buffer that is just longer than the longest packet. As data comes in we shift the bits to across so that new data is always on the end of the buffer. Then we search the start of the buffer for the unique word. If we see the unique word, we try decoding both Horus Binary v1 and v2 - Only one of the checksums should pass, if it does, then we have a valid packet.

This however means that we have to wait the same period of time for v1 packets as we would for the much longer v2 packets.

v1/v2 shortcomings

Now lets summarize some of the shortcomings for v1/v2

• Launch operators require a central authority to bless their callsign ID and custom payload data
• Receiving stations need to regularly phone home to get latest configs
• Custom payload data is rigid and inflexible
• There is latency in decoding smaller packet sizes
• Small size of 30 bytes can limit usefulness for some missions

Additionally the software for decoding Horus Binary had some issues:

• Pypi packages didn’t have wheels, resulting in users having to build their own versions
• The modem itself is a C executable which had to be built separately
• Horus GUI Windows app required reusing a handcrafted DLL, limiting ability to update the modem component
• Limited testing / no testing framework meant a lot of manual testing before releases and changes

While still meeting the constraints listed above, can we do better?

Horus Binary v3 and ASN.1

This is where Horus Binary v3 comes in. Horus Binary v3 is an attempt to address the above issues and has taken months of planning, discussions, development and testing. Most of which has been figuring out which things to compromise on. While apps today are running entire browsers and gigabytes of memory, the development of Horus Binary v3 meant squabbling over single bits, let along bytes.

One thing I wanted was a well defined specification of the binary format. After investigating things such as protobuf, Cap’n Proto and many other encoding schemes I was somewhat surprised to find there’s limited options for unaligned formats. Unaligned means that a field doesn’t need to be whole bytes. A field can start and stop at any bit offset, rather than a multiple of 8. Shifting from a byte aligned to unaligned format was important to the design goals as it let use save bits from fields that would otherwise need as much range in their values. Eventually settled on ASN.1 using it’s Unaligned Packed Encoding Rules (UPER).

ASN.1, or Abstract Syntax Notation One is a standardised way of describing data.

For example I can describe a temperature field like so:

internalTemp   INTEGER (-1023..1023) OPTIONAL

In this example the internalTemp field can have a value from -1023 to 1023, and it is optional.

ASN.1 defines a bunch of encoding rules. We can take the above specification and encode it into XML, JSON, or bits. What’s great about UPER is that it takes into account the size constraints like (-1023..1023), so that the final encoding for that field is just 11 bits long for the data itself. The optional flag is an additional field to mark if the field is actually present. So if internalTemp isn’t sent in the payload, then only a single bit is consumed.

Encoding a value of 123 for internalTemp results in 12 bits:

Optional flag   Number
1               100 0111 1010

From experience we know that several fields are always sent - such as Payload ID, sequence number, time of day, location. But we also know that each payload is different and will may have none, one or multiple of sensors like temperature, voltage, pressure, counters. We can place these fields into our specification and operators can pick and choose what they need.

CustomFieldValues ::= CHOICE {
  horusStr   IA5String (FROM("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_ +/=-.")^SIZE (0..255)), 
  horusInt   SEQUENCE(SIZE(1..4)) OF INTEGER,
  horusReal  SEQUENCE(SIZE(1..4)) OF REAL,
  horusBool  BitFlags
}

AdditionalSensorType ::= SEQUENCE {
  name    IA5String (FROM("abcdefghijklmnopqrstuvwxyz0123456789-")^SIZE (1..20)) OPTIONAL, 
  values  CustomFieldValues OPTIONAL
}
AdditionalSensors ::= SEQUENCE(SIZE(1..4)) OF AdditionalSensorType

Additional to the built in sensor types, we also have an AdditionalSensors field which uses ASN.1’s CHOICE function to allow an operator to pick what kind of data type they require. This could be a REAL, INTEGER, BOOLEAN or a STRING. This allows some amount of self describing without the need of central authority.

Since fields and sensors can be optionally added and removed a payload doesn’t need to send all it’s sensor data down, all the time. While location is important for recovery, sensors can be sent sequentially to fit in packet sizes.

Payload IDs have also been replaced with strings. While this consumes more bytes it allows freedom to develop and launch without having to request a payload ID number. ASN.1 UPER rules allow us to define what characters are allowed, and by doing so this reduces the per character cost to just 6 bits per letter.

To help develop the format and understand the packet length costs we built a tool to visual the ASN.1 encoding. Along with a series of unit tests to make sure the encoding was working ok.

The tool allows for changing the ASN.1 specification and input data, allowing payload developers to make decisions on what data to send when and can be found here. It took a lot of collaboration and thought to figure out good compromises. Adding an optional field always consumes one bit so we want to work out which fields are always sent and which ones might not get sent. If we made every field optional we would waste nearly an entire byte in optional flags.

For each field we want to also constrain the size of that field - this means figuring about the absolute min and max values for each type along with the required resolution. I wish ASN.1 had support for fixed precision.

But 30 bytes still doesn’t get us much data. What if we have a bunch of extra sensors.

Longer packets and checksums

We’ve expanded out the packet size to allow 32, 48, 64, 96 and 128 byte long packets. While we don’t recommend sending 128 byte packets as the transmission time is longgggg, it’s an option for those who need it. The 48 byte packet seems like a really nice middle ground balancing the packet size with additional telemetry.

But this long packet poses a problem with the current solution. The latency would increase significantly if we have to wait for the longest packet size before checking all the combinations.

This is why we’ve flipped around the way the packet is attempted to be decoded. As soon as a particular format has enough bytes to decode, we try to decode it, regardless of where it is in the buffer. By switching to scanning for the unique word as the bits come in we are able to decode all packet sizes with the smallest amount of latency.

There’s still one problem though. If we have a 32 byte packet that could be v2 or v3, how do we work out which decoder to use? Our final change of the telemetry format was to move the checksum to the start of the packet. So we check the checksum at both the start and end of the packet. If the one at the start passes, it’s v3 and if the one at the end passes it’s v2.

Transmitting

It’s all well and good being able to generate these complicated unaligned structures in Python on a machine with gigabbytes of RAM but we need it to run on a repurposed microcontroller. For this we use the ASN1SCC compiler.

This can take an ASN.1 specification and turn it into light weight C code.

// some setup and error code removed for brevity
horusTelemetry testMessage = {
	.payloadCallsign  = "VK3FUR",
	.sequenceNumber = 2,
	.timeOfDaySeconds  = 30,
	.latitude = 90,
	.longitude = 90,
	.altitudeMeters = 1000,
};
  horusTelemetry_Encode(&testMessage,&encodedMessage,&errCode,true);

The ASN1SCC is pretty light weight compared to other tools like asn1c and asn1tools. We saw an increase of roughly 6 kilobytes to the compiled output.

Two problems with ASN1SCC is that it’s opinonated about space vehicles. This means that extension markers are currently out of scope and not supported but we were able to work around that placing a dummy optional field. The other is that ASN1SCC assumes you might build a packet that is as long as the longest possible - causing excessive memory usage. We worked around this by building our own assert handler and allowing smaller buffer sizes.

Packaging and tests

There’s three main way Horus Binary v3 is received. Using horusdemodlib’s horus_demod command in a shell script, webhorus in the browser and the horus-gui desktop application.

Horus-gui uses horusdmeodlib which demodulates the signal using libhorus C library. The result is that for both horus_demod shell script method and horus-gui options, C code needs to be compiled using cmake. This means extra compiler tools to install and extra steps for the user to follow.

An additional concern that by using ctypes to access the library in Python there is risk in programming errors causing subtle memory corruption bugs that are hard to catch. We have to carefully define all the arg and return types for each function call like so.

self.c_lib.horus_set_freq_est_limits.argtype = [
            POINTER(c_ubyte),
            c_float,
            c_float,
        ]
self.c_lib.horus_get_max_demod_in.restype = c_int

To resolve these issues we made the change from ctypes to cffi. This makes Python in charge of the compiling, and automates the creation of a library which handles the args and return types correctly. Additionally we converted the C based horus_demod command to Python.

With Python now in charge of compiling we could start using cibuildwheel to automate making Python Wheels (these are precompiled ready to go packages that are built for each desired Python version and architecture). This means the majority of our users do not need to compile to install and use horusdemodlib and simplifies the build process for horus-gui.

A user can now do pipx install horusdemodlib to install decoding and uploading utilities². No compiling needed in most cases.

While there were a few manual test scripts, these weren’t run as part of any automated workflow, nor did they use any testing framework. These were updated, added to and enhanced to run using Python’s unittest and report as part of GitHub actions workflows.

While not exhaustive, it gives us a lot more confidence prior to release.

Fin.

And that’s it. horusdemodlib and related apps were released yesterday. There’s been numerous test flights leading up to the release (many thanks to everyone testing, both from the transmitting side, and the receiving side). While there’s likely bugs and quirks, I’m fairly confident that V3 is good move forward for the project.

There’s still some things we want to work towards, such as easier receiving CLI app design, Debian packaging and possibly some micro python payloads. First up however, I’m taking a nap.

Further reading

• Horusdemodlib GitHub Repo
• Horus Binary v3 ASN.1 README
• Horus GUI
• AREG Project Horus
• SondeHub Amateur

Amateur Radio Experimenters Group (AREG) decided to plan a winter balloon launch. Something that they typically don’t do due to the unpredictable weather and cloud cover.

One of the ideas for the winter launch was to be a crossband FM repeater and the increased winds to Victoria would mean a larger coverage area for possible interstate communication. The other (and more interesting to me) benefit was real world testing of the webhorus and webwenet sites which simplifies decoding of both balloon protocols. While others had used the app and reported success, I had only ever tested it in my lab. It’s a bit of a weird experience to have never used my own application in a production setting.

The first possible launch date got pushed back due to bad weather. The second launch window wasn’t looking much better and if that was cancelled it was unlikely that the launch would go ahead until much later in the year.

The predictions were showing a longer distance than we could travel (legally) during the flight time - meaning that we likely wouldn’t be near the landing site on touch down. Chance of recovery would be lower.

Keen to still test my software I ask Mark if we I provided a small low cost payload if we could fly something.

I rushed together parts I had for a Wenet transmitter. Given the main payloads weren’t flying maybe I should try something a bit more experimental in nature.

As AREG is in Adelaide and I’m in Melbourne it also meant driving the 8ish hours across. It was agreed that I would build up the Pi / transmitter and the box, antenna and camera would be installed when I arrived - leaving only a few hours in the evening for that integration to happen before launch the next day.

I2S PCM Audio

While developing webwenet I learnt a lot more about how Wenet worked. One thing that really stood out was that Wenet receivers need to remove RS232 framing from the data before decoding the packets. This initially struck me as odd. What does RS232 have to do with this protocol.

Today most Wenet transmitters use a RFM98W, usually on some LoRa shield/package. The UART pin from the Raspberry Pi is connected to DIO2. The chip is configured for a 2-FSK direct mode, bypassing pretty much all the smarts of the chip itself. If the DIO2 pin is high it transmits one tone, and if it is low it transmits the other.

Since the UART is used to transmit the packet that means that RS232 framing is also transmitted.

I had a quick look at the Raspberry Pi datasheet for the UART and didn’t really see a obvisous way of disabling the start and stop bits. It would be possible to bit bang data out - but timing is important for us on RF so I didn’t explore this option.

The Raspberry Pi however has a ton of different interfaces. Maybe another one would work better without the framing. My first thought was - maybe we could just bit bang a GPIO. The Pi is pretty fast and we can use some kernel features to make that work.

My next thought was along the lines of SPI, but Droppy reminded me that the Pi has I2S output - which is IMHO the classic bit banging target.

Well before the Horus flight was even planned I prototyped an I2S version which emulated the the UART (basically adding back in the RS232 framing) to compare in the lab.

Now the proper way to implement this is probably to access the I2S interface directly, however easiest approach for me to use the RaspberryPi_I2S_Master device tree driver which adds a generic sound device. This is expected to be tied to a audio chip - but we just ignore that part. From the code point of view we import alsaaudio and generate the correct audio frames.

Some complications show up though. The linux simple-audio-card driver seems to have fixed set of sample rates and channels. I don’t think there’s a specific reason this needs to be the case, but without building a new kernel module it means a little bit of extra work. Now channel wise it doesn’t matter so much, we send the samples irrespective of which channel is selected (that pin gets ignored entirely) - but we do have to factor the channels into our data rate.

So with only hand full of sample rates to choose from how do we get the bit rate we desire. At startup we calculate the desired RF bitrate and then work out multiples of the audio sample rates.

For example - if the audio sample rate is 48000, the number of channels is 2, 16 bit audio width and the desired RF baud rate is 96000, then the audio bit rate is 1536000, then we divide by 8 bits per byte - 1536000/8 = 2 bytes. So for every “1” bit we send two bytes - 0xffff and for every “0” bit we send 0x0000.

In testing however I noticed that my transmitter just wouldn’t work when running the software. It turns out the Wenet TX software had an LED configured for the same GPIO that was used for I2S PCM out on the Pi. Annoyingly when this GPIO is set the Pi had to be rebooted before the I2S output would work again!

To make use of the advantages of I2S approach I needed to write new transmitter code which didn’t include the RS232 framing, update the existing Wenet receiver software, and my webwenet code. This was a big undergoing so close to the flight deadline, but we managed to do it.

By removing the RS232 framing we remove the 20% overhead from start and stop bits. This allowed us to pick a lower RF bandwidth while still being slightly faster than the original system.

Modulation testing

It’s all fine doing this in practice and testing locally where SNR is high, but we need to make sure that both the modulator and demodulator is working correctly. Subtle errors in things like the parity code, timing mistakes or off by one errors won’t be apparent until low SNR.

To do this Mark built some benchmarking scripts. These take a high SNR recording, then generate lower SNR samples. The lower SNR samples are feed back into the demodulator and a count of the packets decoded is taken. With the low density parity check (LDPC) and the quality of the modem we expect a fairly sharp fall off. The nice thing about these scripts is that they normalise for Eb/N0 (SNR per bit). This means that even though our baud rate is different we can still compare the new I2S method to the old UART method.

When we run this I2S we see similar dB threshold where all the packets are received however there’s a bit of a weird slope. We expect that due to the parity check that either you receive a packet or don’t, so the dB difference between that threshold point should be small. This lead us to believe that the testing wasn’t working right.

Now one thing that we noted when switching to I2S is that the UART method didn’t include any sort of scrambling or whitening. Scrambling is used to ensure that there isn’t a long run of zeroes or ones which could cause the modem to loose timing and become unsynchronised. For the UART mode this wasn’t a big concern because the RS232 framing meant that the start and stop bits would always cause a 1 and 0. Switching naively to I2S meant that we lost this free bit flip.

The theory I have at the moment is that the long runs of 1s or 0s in the I2S approach breaks the normalisation scripts. Adding in scrambling we get a much sharper cut off as expected.

Dual mode

While I was pretty confident in the I2S approach we wanted to be able to compare the two modes in a real world setting. A normal person might just fly two payloads or have two transmitters. Instead however I decided that it was likely possible to have the software switch between the two modes.

To do this a soldered diodes to the output pins on the Pi and feed both UART and I2S to the RFM98W module. With the hardware done, we need to look at the software. The first problem is that the UART transmitter needs to disable its output when its not active. To do this we use the break_condition attribute in pySerial. I2S luckily sits low normally so nothing needed to be done there. Finally we need to switch between the two modes in the transmitting software - which it was never really built for. I hacked in the functionality to have a list of radio modules which are cycled only when the radio is in an idle state after a period of time.

The result was that the radio switched between modes roughly every 1-2 pictures.

Testing deadlines

I moved my code over to the target Pi Zero 2 W. Went to plug in my PiCam only to find the connector is different between the Zero and the normal Pi. From a “launching the balloon” point of view it wasn’t going to be a problem as the camera was being supplied by Mark however from a testing point of view it meant I wasn’t going to be able to test that the camera functions. I rushed off an order to Amazon for a camera module that did have the correct cable.

In the meantime I tested without the camera module. I had some extremely weird issues. File descriptors were being opened and after about an hour the software would crash due to reaching the ulimit. Debugging the issue and I couldn’t determine what was causing it - I thought it was caused by swapping between the two radio modes, but that didn’t seem to be the case.

I tried a bunch of Python debugging tools to see the cause but couldn’t nail it down to any Python code. It seemed to be occurring due to the picamera2 Python module. It seems that file descriptors were being leaked due to the lack of camera connected - but I wasn’t entirely certain. To alleviate the problem I bumped up the max number of files open in ulimits however it turns out that select is limited to 1024 file descriptors anyway - so that option didn’t work.

I quickly added a shell script to check for open file descriptor count and put into the watchdog.d config. If it got too many fds it would reboot and everything would be fine. It’ll do.

Integration woes

Arriving in Adelaide I passed the payload to Mark who quickly fitted it into the box, added the camera and we fired it up.

Two issues presented us. The GPS didn’t stay locked / struggled to lock. I had seen some of these issues with my testing - we’re not entirely sure why my uBlox 6 chip wasn’t working correctly as ubxtool showed correct data. Current suspicion is that the code was written for uBlox 7 and isn’t happy with some of the data. The other factor is that I was using soft serial (as UART was taken for I2S transmitter). After some quick debugging we opt’d for replacing it with a known good USB GPS module.

More concerning was that with the new camera connected the Pi rebooted shortly after initial boot up. We believe this was a kernel panic but with no swap setup to do coredumps we don’t really have any logs of that. I believe this was to do with the PiCamera 3 module as I didn’t see the reboot happen in my testing.

The reboot didn’t happen often and the system started up again just fine. But it’s not something I really like to see before launch. The picamera2 github is full of issues of people having random issues with the library. I even saw some issues where Python crashed somewhere that it practically couldn’t, and I think this was also caused from the picamera2 library - my guess is that it lacks thread safety and incorrect/lack of locking.

I put in some extra error handling. We did a few tests and it seemed fairly stable, even if it did reboot. However it exposed a known limitation in webwenet. Wenet protocol sends images with a byte for image header, followed by a byte for an image id. Restarting the payload resets the image id. When webwenet receives an image is combines all the packets for an image based on the image id. With the image id reseting the images were being updated rather than replaced in the UI. So a last minute patch/hack was added so that users wouldn’t get confused if the image id was reset.

Launch day

We arrived at the launch site. The car was prepped for receive mode and the balloon was filled. Up until now we hadn’t really decided if we wanted to chase the balloon, or if we wanted to get to a higher location to have a better chance of receiving as much data as possible. We opt’d for the high location and would attempt recovery on our drive home the following day.

Many hands were required to handle the balloon during filling and tying off, but eventually it was filled and launched.

Droppy drove while Alex and I monitored the balloon and receivers. One of the things I wanted to test with webwenet was around lowering the entry requirements for receiving. What would be the minimal equipment needed to receive?

For that I went with:

• LNA4ALL - €23 + shipping
• RTL-SDR $33.95 USD
• AliExpress 7 Element Yagi - $93 AUD
• AliExpress 3dbi “short” vertical antenna - $15 AUD

The LNA4ALL was chosen as it can be powered directly from the RTL-SDR using the Bias-T option. The LNA4ALL does need a small modification to allow this.

I created a few little brackets for our cars roof rack that mounted the LNA close to the antenna.

The small vertical antenna is used while driving as it provides low gain. A higher gain antenna would result it a more narrow beam which isn’t very useful if the balloon is high up.

Once we arrived at the high location we switched to the yagi antenna which provided a reasonably good SNR throughout the rest of the flight.

This setup worked fairly well and was pretty on par with some of the bigger setups we saw on the day.

Fun with phones

One of the fun things with webwenet and webhorus is that you can load it up on mobile phone browsers. This meant that we hooked up a mobile phone on a many-element yagi - something that just seems extremely ridiculous.

Pi Camera 3 Focus issues

As images started streaming in it was obvious that a long running issue with the Pi Camera 3 had struck us. Out of focus images. Current suspicion is that the payload is moving too much for the Pi Camera 3 to obtain proper focus. Which is a shame because some of these pictures would have been stunning. Regardless I still think they look pretty good.

The highest picture was recorded at 20,116m.

Landing and recovery

The balloon landed over 250km away. The last reported altitude was 1,083m.

Being so high up for the last reported location leads to a very large search area. Local ground winds play a huge part in final landing location, so it was going to be a challenge to recover. Probably unlikely given that the transmitters batteries would now be flat and uncertainty regarding if the balloon still had a red parachute to spot.

Arriving at the predicted landing spot, the farmland covered the repeating dunes. Luckily being winter the fields were bare. We stopped on the second dune near our suspected landing location and walked across to the next. Not even sure which field it had landed in we kept scouting around. Part way up the next dune I spotted in the distance a red patch with two white items near it. It was a long way away but given it was unlikely that the field would otherwise have something like that in the middle of it I was pretty certain that it was the payload.

Walking across the field I arrived at the landing site. It was a surprise to recover the payloads with the data we had, let alone find them so easily. I guess one advantage to winter launches is empty fields. Enjoy the time lapse created from the remaining battery on the image transmitter payload.

After bringing the payload back we opened it up and turned it back on to get a group photo before heading back home.

The final verdict on I2S

It performed no worse than the UART version. The theoretical performance increase of the lower RF bandwidth requirement isn’t actually much - to the point that it’s in the testing noise. We do know that the data rate is slightly faster though. The other advantage is that it frees up the Raspberry Pi hardware UART for other tasks. It sounds like AREG will be flying I2S Wenet going forward after the success of the dual mode payload.

Thanks

Many thanks to AREG, Mark, Droppy, Alex and all the receivers who let me fly this payload, help with recovery and put up with my bullshit.