Software apps and online services
Hand tools and fabrication machines
Please note that starting from version 9 the software makes usage of latest Skyfield version (1.3). You can install Skyfield using pip install skyfield
It is no longer necessary to install using pip install skyfield==0.6.1
I also fixed a bug about proper management of negative (southern) longitude values.
Many thanks to Dave Typinski for the additions/corrections/phase plane probability map I added the various edits to paragraphs "Which kind of signals could I receive ?", "A bit of scientific background", "About CML-Io Phase Plane Probabilities".
I updated the software used in this project to use a different FFT library and different file format for improved performance and reduced storage space.
New software is available on github here
This new version makes use of a modified version of rtl-power-fftw by Klemen Blokar and Andrej Lajovic. The new sources are available here
Old version of the software is available here
All the software has been built and tested both on Raspberry PI 2 and 3.
It runs well also on a Debian Jessie virtual machine.
rtl_power_fftw can be compiled also on Windows but I did not try this yet.
Amateur radio-astronomers built several devices in order to receive, listen and/or display signals from outer space sources, many use custom made radio receivers, others use HAM radio receivers. Recently, various experiments have been done using the cheap rtl-sdr dongles (radio receivers originally meant to listen radio and watch TV on your laptop).
With this project I want to try receiving radio emissions from the Sun or Jupiter and its satellite Io. I want to build an automated radio station, cloud connected as an IoT device that any amateur could easily replicate, joining the project and forming a collaborative, open science effort to easily share the spectrograms of the received emissions.
It is very easy to receive Radio Frequency storms from the Sun, but also Jupiter produces strong RF emissions. You can receive radio emissions from these sources on different frequencies. The upper part of the HF band ( between 10 and 30 MHz ) is often the target of scientific monitoring for both Sun and Jupiter. Many other frequencies can be monitored up to 1.7 GHz with these radio dongles but I will focus on the HF band with this project.
Jupiter decametric radio emission, first discovered by Burke and Franklin in 1955, can be received between 15 and 30 MHz with a modest antenna array. The Earth's atmosphere attenuates the emission progressively more the further one looks below 15 MHz. Jupiter’s magnetic field strength limits its decametric emission to a maximum of slightly under 40 MHz, but Jovian emission also gets weaker the higher one goes in frequency. Most Jovian emission above 30 MHz is weak, requiring large antenna arrays to see it.
You can read the whole story of the discovery of Jupiter’s radio storms on the wonderful Radio Jove NASA website . On the same website you will find the description of a very good dedicated radio receiver on 20.1 MHz .
The tuning frequency is not very critical since the radio storms can be received across most of the HF spectrum. The emissions are produced by the interaction of charged particles with the Jovian magnetic field. The Jovian magnetic field is about 20,000 times stronger that Earth’s. As with Earth’s aurorae, the Jovian magnetic field is responsible for aurorae formation in Jupiter’s upper atmosphere (like pictured below by Hubble). The Jupiter emission observed in the HF band does not come from the aurorae, but from a region around 20,000 km above the visible cloud layer.
The relative geometry between Jupiter, its satellite Io, and Earth has been observed to have a modulating effect on the emission, making it more or less likely to be observed at different times. There's a probability distribution chart that you will find in various scientific papers that has been built based on years of data logging since 1957. Data is still being collected and the probability parameters updated.
It is being used by some programs to forecast and display the best listening periods. Here is an example:
The plot above, called a CML-Io Phase Plane Probability Map (or just “phase plane” for short), is an adaptation from the probability map used by Radio Jupiter Pro.
Central Meridian Longitude tells us what part of Jupiter is facing Earth, while Io Phase tells us where Io is in its orbit relative to superior geocentric conjunction (the point in its orbit furthest away from Earth, directly behind Jupiter).
The zone boundaries (with the exception of Io-D) are marked according to the University of Florida Radio Observatory (UFRO) definitions; see https://ufro.astro.ufl.edu/decframe.htm . UFRO does not define an Io-D zone, so the Io-D zone is marked according to the definition provided by Carr, et al. in “Physics of the Jovian Magnetosphere” (1983).
The Io-CML phase plane image above attempts to depict the relative probability of receiving Jovian emissions at 20.1 MHz. This is done by first making an average of probability data generated from observations made at 18, 20 and 22 MHz at the University of Florida Radio Observatory (UFRO) from 1957 to 1994. The resulting average is then scaled so that the peak probability, in the Io-B source region, becomes 100% relative probability. The probability of observing Jovian emissions is affected by many variables. Some of these are the observing frequency, transparency of the earth's ionosphere, duration of the observing session, antenna gain, receiver sensitivity, galactic background noise level, man-made noise level, position of Jupiter relative to the Sun, and the jovicentric declination of Earth. While this image is a useful guide for the Jove observer, it cannot be used to predict events with absolute certainty.
Experience with small antenna arrays and swept frequency spectrographs shows that 100% relative probability is perhaps 50% absolute probability. That means even for the high-probability Io-B zone, one has only a 50% chance of observing Jovian emission when Jupiter, Io, and Earth are in such a configuration.
Thanks to Drs. Chuck Higgins, Francisco Reyes, and James Thieman, for their assistance in making this UFRO data available, and to Dave Typinski for generating the phase plane graphic image above.
Here is the same phase plane and probability data represented in 3D:
3D plot above is from Figure 1.4 in Higgins, A New Determination of Jupiter's Radio Rotation Period, University of Florida (dissertation) (1996). It represents observations made at the University of Florida Radio Observatory from 1957 to 1994 at 18, 20, and 22 MHz.
You can find additional information on Jupiter radio emissions in this book by Jeff Lashley: The Radio Sky and How to Observe It
On windows you can use Radio Jupiter Pro:
On your Raspberry I'm providing a python script named jpredict.py (see my github code repository) that will output the predictions in text format. It's a port to python of an older QBasic program whose original source can be found at this web page together with good explanations about the theory and instructions to build a folded dipole antenna.
You can find the python porting here on Stackoverflow and here on github: https://github.com/adamk/Jovian-radio-emission-flagger
Warning: that version will give general predictions, not specific to your observing position. So, if Jupiter is below the horizon for you when the program predicts emissions... it will be actually impossible for you to receive them.
I heavily modified that version adding the wonderful python package SkyField by Brandon Rhodes, to calculate event predictions for your geographical position, you can specify latitude and longitude in file radioConfig.py .
Here below, you can see an example output of jpredict.py where for each UTC date time, the predicted event type is shown (if any):
An example of a solar flare radio burst and fadeout as received on Earth (Culgoora Observatory, Australia):
As I wrote in the introduction, in this project I'm focusing on a different approach to radio data collection: I will use an RTL-SDR usb dongle : It is essentially a consumer device originally designed as a TV tuner for DVB-T to be used on computers through a USB port.
Recently, a new set of software driver and programs have been written and upgraded by various people. I'm using the version that has been written by keenerd . This sw allows easy tuning on various frequencies, saving on disk the strength of the received signals. Plotting the signal strength data vs time you obtain a spectrogram like this:
Signal strength is calculated by program rtl_power performing FFT transformations on the digital stream of data that comes from the tuner via the USB port. Different colours are assigned to the signal strength values plotted using a specified palette visible on the right.
Note: version 2 of my python programs now use a modified version of rtl-power-fftw (see update note at top of this project page).
I would like to keep the radio and data processor switched on for several hours per night (or potentially always). Using a Raspberry PI 2 I could have a low power consumption (about 2 Watts), so I would perform all the following directly on the R-PI:
- connect directly the RTL-SDR USB dongle receiver to the R-PI
- do the radio spectrum scan and data writing directly on the R-PI microSD
- produce the spectrum image on-board from that data
- publish on the web the resulting image and/or data sending to the Amazon AWS cloud for centralized storage
- send push notifications (MQTT) to other amateur observers for open science data sharing
- I would envision the adoption of this simple system by several amateur radio-astronomers that could form a network of distributed radio stations, connected as IoT devices to the central repository
- free usage of the publishing system should allow easy access to historical data stored on AWS S3
- the message based notification system should facilitate the integration with other systems and/or further study / processing
- the proposed design should work for most of the usual radio-astronomy frequencies. I will use the recommended 18-24 MHz frequency range, probably the most used one for Jupiter-Io radio storms.
The overall project diagram is here:
Why ? Because a wire antenna cut for 20.1 MHz has its own size and needs to be positioned properly. If you don't have the necessary space you should consider an alternative frequency.
A very good example dipole antenna design can be found on the NASA radiojove website here together with important information about Jupiter position, visibility of the source from your site, etc. You can find the wire length for the dipole calculated inside the pdf: it's 7.09 meters , consider the length of the rope guylines for the masts to have an idea of the free ground space you need to setup the antenna.
Important: the only difference in my usage of this antenna design is to use an SMA male connector on the receiver end of the coax cable. This because the upconverter and many of the SDR radios use that connector type.
Another constraint is given by Radio Frequency Interference. I would suggest you start doing a "survey" of your selected antenna site to see how much RFI is present and on which frequencies. You can do this connecting the RTL-SDR to your laptop and using free sw like SDR# to watch/listen the whole spectrum. If you see that too much radio interference is present, you can choose a different location for your antenna, add proper filters between the antenna and the receiver or change the target frequency of your research. In some cases you simply cannot proceed with such a project: it can happen in highly populated urban or industrial areas. As they say... your mileage may vary.
When I run the first scans, I've noticed a lot of radio interferences. I've used the instructions on this web page to reduce most of the RFI I was getting in my receiver / logged data when using the first dongle I purchased (with plastic case, then I purchased the latest with metal case and TCXO).
This project uses the following hardware (see also the hardware list from the top menu):
- Raspberry PI 2 with its own power source and network cable
- microSD 16 GB, class 10 minimum, for the OS and files storage
- RTL-SDR dongle based on R820T tuner chip (many available on eBay, I've bought the latest by rtl-sdr blog because the already have a shielding metal enclosure and a more stable clock source)
- up-converter for HF frequencies (see text below)
- low-noise 5V power source for the up-converter (I've used an old custom made one, you can buy/build one similar, will add the circuit diagram)
- coaxial cable as specified in the antenna manual above
- RF connectors (SMA and BNC) (many available on eBay)
- some metal boxes to shield the up-converter (and the dongle if plastic made) from RFI
Some pictures of these hardware items:
The RTL-SDR can tune to a wide set of frequencies, starting from about 24 MHz. Since we are interested also in frequencies below 24 MHz, we need an up-converter. It's a simple circuit that "shifts" the RF up by a certain amount so that becomes receivable by these dongles. You add it between the antenna and the RTL-SDR. In my case I've used one that shifts by 125 MHz so, when you tune the dongle to 145.1 MHz you are actually tuning it to 20.1 MHz .
Free SW like SDR# has an option to manage that easily during your survey / tests:
Important: should you decide to work at a different center frequency, higher than 24 MHz you can obviously avoid purchasing and using an up-converter. Find below a picture of my SDR UP-100 made by Adam 9A4QV. Unfortunately it's sold-out but you can purchase any other like this for example.
Please note that the upconverter usually has an input low-pass filter, in this case allowing input frequencies in the range 1.8-55 MHz. This is important because it can constraint your research frequency range but is also protecting your receiver from the powerful FM stations working at 88-108 MHz that can be really disturbing depending on your position.
Please note that I used SDR# only for my surveys, it's not needed for running this project that is based on R-PI-2 + AWS cloud.
The radio jove antenna manual explains how to buildRF cables soldering connectors. Should you encounter any difficulty, I suggest you search youtube for videos: you will find nice instructions about soldering RF connectors.
Please follow the instructions below in the sequence they are presented.
Some of these commands will ask you confirmation: reply yes when requested.
If you already have the latest Raspbian Jessie OS ready on a suitable microSD you can skip to the next block "Prepare the development environment".
Download the OS image file from here: https://www.raspberrypi.org/downloads/
unzip and write to microSD using win32diskimager following the Raspberry Pi official instructions
insert the microsd in the R-PI slot and switch on / boot the R-PI
connect using Putty or similar terminal client using the default credentials (user: pi password: raspberry)
extend the micro SD partition to full size using the raspi-config tool menu:
(please note that starting from Raspbian Jessie version May 2016 the file system will be automatically expanded on first boot)
then select exit and reboot using this command:
set the language / keyboard / time zone to your preferences using again:
We will compile the radio software from keenerd sources, so we first prepare the development environment with some tools/libraries. Enter the following:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install cmake
sudo apt-get install libusb-1.0-0-dev
git clone https://github.com/keenerd/rtl-sdr.git
in this way you prepared the development environment and downloaded the open source sw tools for the RTL-SDR radio that will be used to run the scans.
Build the software using the following commands:
cmake ../ -DINSTALL_UDEV_RULES=ON
sudo make install
at this point the sw has been built and installed.
Change directory so that we can create a file to avoid issues with the device drivers:
create this file:
sudo nano no-rtl.conf
and now enter these 3 lines:
blacklist dvb_usb_rtl28xxu blacklist rtl2832 blacklist rtl2830
and then to save hit ctrl-o and then ctrl-x to exit
If your rtl-sdr dongle was already connected to the R-Pi usb port when you booted up, you need to remove the driver currently loaded:
sudo rmmod dvb_usb_rtl28xxu rtl2832
Go back to the main folder:
now connect the dongle to one of the R-PI usb ports and enter this command to run a first test:
this will test the possible tuning range and packet loss if any. It will automatically find the dongle device and start the test. If you get errors please check the previous steps.
Ever since we want to avoid producing additional RFI, we should disable the R-PI-3 WiFi and bluetooth. We will connect exclusively using an ethernet cable with ferrites possibly at both ends.
In order to disable both WiFi and Bluetooth, enter the following command:
sudo nano /etc/modprobe.d/raspi-blacklist.conf
then add the following lines:
#wifi blacklist brcmfmac blacklist brcmutil #bt blacklist btbcm blacklist hci_uart
and then to save hit ctrl-o and then ctrl-x to exit
Reboot your R-PI entering
sudo shutdown -r now
At this point you will be able to connect using the cabled network that is enabled by default.
Note: first version of the radio station python programs used GNUPLOT to produce the spectrograms. Latest version does no longer use GNUPLOT and instead, makes use of matplotlib, a python module that is smaller to install.
So you can install matplotlib with the following command:
sudo pip install matplotlib
We will also use a python based astronomy package named SkyField that has some other code dependencies. It will take some time to compile numpy, be patient (on Raspberry PI 3 it's already available within the OS image):
sudo apt-get install python-dev
sudo pip install numpy
these will take some time to complete and display several warnings during compile ( on R-PI-3 3'50" for numexpr ) :
sudo pip install numexpr
sudo pip install pytz
then install skyfield, pyephem ( on R-PI-3 1'27" ) and the AWS Python modules:
Please note that starting from version 9 the software makes usage of latest Skyfield version (1.3). You can install Skyfield using
pip install skyfield
(It is no longer necessary to install using pip install skyfield==0.6.1 )
sudo pip install pyephem
sudo pip install boto
Go back to the home folder to install the MQTT module for python and the datetime tool to force clock sync with NTP servers (more on this few lines later).
sudo pip install paho-mqtt
sudo apt-get install ntpdate
The Raspbian image comes with NTP daemon already installed with default configuration. This means that you will get the date/time updated within few minutes from boot. Without going too much in detail, let's say that you can configure your preferred servers so that it can reach them with the shortest possible network delay. I have the Italian official atomic clock source at 40 Km so I will configure its 2 NTP servers on the R-PI. You can search the web to find out yours closest one. Why I'm doing this ? Having the data properly time stamped is important when you have to compare your data with that produced by others and having it precise enough is a good start.
enter the following:
sudo nano /etc/ntp.conf
scroll to the first "server" statement and add one or more lines like these:
server ntp1.inrim.it server ntp2.inrim.it
We now do a similar config for ntpdate:
sudo nano /etc/default/ntpdate
add these 2 servers :
Now install one more package used by my python programs ( on R-PI-3 3'28") :
sudo apt-get install python-matplotlib
make sure you are positioned in your home folder:
then retrieve my custom scripts and python programs:
git clone https://github.com/mariocannistra/radio-astronomy-fftw.git
Make a copy of radioConfig-example.py as radioConfig.py:
mv radioConfig-example.py radioConfig.py
Configure your position, scan parameters and other preferences editing radioConfig.py:
sudo nano radioConfig.py
you will see three groups of settings: radio scan, position of observatory, AWS config:
# scan parameters configured here # are used by all the scripts freqCenter = 21000000 # example: if you want to scan from 17 to 25 MHz you enter here 21000000 as center frequency and 8000000 below as bandwidth freqBandwidth = 8000000 upconvFreqHz = 125000000 # this is your upconverter offset frequency in Hz, set to 0 if you're not using an upconverter # maximum value successfully tested on the R-PI-3 : 2800000 rtlSampleRateHz = 2000000 # specify one of the two values and set the other to 0 totalFFTbins = 4000 # this is the total number of bins available on a scan. will be divided by number of necessary hops and is affected by crop percenteage binSizeHz = 0 # this is the FFT bin size in Hz, lower values will give you more detail in spectrograms, but larger files to process on the R-PI # Specify the percentage of bins to be discarded on each scan. # It will actually be half on each side of a single scan. cropPercentage = 30 # specify one of the two values and set the other to 0 integrationIntervalSec = 0.5 # see rtl_power_fftw documentation, this is the integration time in seconds. can be < 1 sec integrationScans = 0 # see rtl_power_fftw documentation, this is the number of scans that will be averaged for integration purposes. Allows to integrate for less than 1 second. subtractBaseline = False # when true a previously recorded baseline scan will be subtracted from every scan before saving to disk gain = 30 # maximum gain is about 49, can be too much in certain positions with strong interfering sources linearPower = False # flag to calculate linear power values instead of logarithmic tunerOffsetPPM = 0 # see rtl_power documentation, i can use 0 on my TCXO based dongle, cab be anything from 0 to 60 or more depending on the dongle (use kalibrate to find it out) dataGatheringDurationMin = 30 # duration of single scan in minutes. 30 minutes is the suggested duration for better chart and limit on memory available on R-PI sessionDurationMin = 600 # overall session duration in minutes 10 hours = 10 * 60 = 600 scanTarget = "Jupiter" # this string will be used to create a subfolder under which all sessions spectrograms will be stored (one subfolder per each date) # the following observer position is used by Jupiter-Io radio emission # prediction utility jpredict.py to calculate Jupiter apparent position / visibility # for your site: stationID = "myObservatoryNameOrAcronym" stationTimezone = "Europe/Rome" stationLat = "00.00000 X" # enter your station latitude here with this string format "00.00000 X" where X is either N or S (North or Sud) stationLon = "00.00000 X" # enter your station longitude here with this string format "00.00000 X" where X is either E or W (East or West) stationElev = 0 # enter your station elevation in meters (above sea level) # predictdays = 5 includeonlyiorelated = False calcinterval = 15 prefLocal = True # False = output in UTC - True = output in local observatory time zone # plotWaterfall = True generateThumbs = False uploadToS3 = False # if you want to upload your scans sendIoTmsg = False # if you want to send notifications. Please, get in touch with me to get the certificates in order to send mqtt notifications # do not change the following lines: awshost = "data.iot.eu-west-1.amazonaws.com" awsport = 8883 clientId = "mcawsthings-test2" thingName = "mcawsthings-test2" caPath = "./aws-iot-rootCA.crt" certPath = "./cert.pem" keyPath = "./privkey.pem"
Now download the JPL ephemerides files using the following command. It will download 2 files for a total of 1 GB to your R-PI (relax, it will do this only one time)
As an alternative, you can download them from http://naif.jpl.nasa.gov/ and then copy them to your R-PI via sftp or WinSCP. These are the direct urls for the 2 files:
In this case download them to the same /home/pi/radio-astronomy folder and then run in any case the following to test that the SkyField based programs can work properly. The program will find the files already there and will just output some test information:
In order to send notifications using MQTT, you need 3 IoT certificate files to be in the same folder of my python programs. If you wish to participate in the cloud based shared experience just get in touch with me and I will send you those 3 files. If you don't want to send notifications just edit radioConfig.py to leave sendIoTmsg = False .
In the family picture below, you can see all the devices. The DSO on the background is not mandatory, I've used that to run some checks for a possible extension of this project.
There are more pictures, click on the right to go through all of them.
Following the overall diagram, you see there is a chain of devices going from the antenna to your internet router:
Connecting things together is simple following the list below:
- connect your coaxial cable "receiver-end" SMA connector to the input SMA female of the upconverter
- connect a shorter (less than 1 meter) cable with SMA males from the upconverter output to the RTL-SDR (most of the plastic case R820T dongles have an MCX female connector, so use an SMA to MCX adapter here; if you bought a dongle with SMA female... you are ready to go)
- connect the USB extension cable female to the dongle USB male
- connect the USB extension cable male to one of the Raspberry PI female ports
- connect a network cable between the Raspberry PI 2 ethernet port and a free port of your internet router
- connect the 5 V power supply to the upconverter and switch it on
- connect the R-PI power supply to the R-PI micro-usb and switch it on
At this point you will be ready to connect to your R-PI using a tty console program like Putty if you are working on Windows or just your terminal session if you are working on a Linux machine.
Here are some photos of my old custom power source. It uses a transformer plus the usual rectifying bridge, condensers and a 7805 regulator. very very low ripple, high stability no RFI (like I'm getting instead with various switching power sources like phone chargers and similar).
You can use any good quality power supply with 5 V and 150 mA output (the upconverter and any other experimental addition like a LNA will easily stay below 100 mA. I'm not going in detail on this since it's not core to this project.
Please have a look at the radioConfig.py file where you will find all settings explained.
They are quite simple and you will find it really useful to have them all in the same place before uploading the file to the R-PI (or editing it there) and starting a session with bash run.sh
I took quite some time to build and develop this project and recently started running scan sessions during the night. These are some of my first outputs:
A scan session is started by entering
This command will force the already mentioned NTP time sync and then start the main loop managed by doscan.py . This program will use the radioConfig.py parameters to run the rtl_power tool and to asynchronously plot the spectrograms.
During the scan sessions, the R-PI will save on the microSD the output of rtl_power in chunks of 30 minutes (default config that you can change).
This files are named using most of the config parameters, for example:
UTC20160131025223 is utc format date and time (yyyy mm dd hh mm ss)
Jupiter is the target of your scan
17.000M-25.000M is the range of frequencies on which the signal intensity has been calculated with FFT and stored in the csv
4000 is the width in Hz of each of the output FFT bins
i1s represents the integration time of 1 second ( see rtl_power doc page for detailed explanation: http://kmkeen.com/rtl-power/ )
g35 is the gain used by the receiver. Acceptable range is 1 to 49 . Using 0 is not suggested since then dongle will switch to automatic gain (= AGC on) and that is not used for this kind of measurements since it alters mesurements.
e30m is the duration of the scan (30 minutes in this case)
As soon as the scan duration elapses, the .csv file is passed to postprocess.py that will run in parallel to prepare the file and plot it using gnuplot. The main execution will not wait for it complete and will immediately launch a new scan with rtl_power. This loop of independent operations will continue for all the time you configured in radioConfig.py . For example, with the following settings it will last for 10 hours (10*60 mins = 600) and will produce 20 files each with 30 mins of data.
dataGatheringDurationMin = 30
sessionDurationMin = 600
At the end of the scan session, another program will be run to determine the overall range of signal strengths received during the whole session. Its output will be stored in 2 files: dbminmax.txt and session-overview.png . The first contains two rows of text with just the maximum and minimum of the whole session. The second contains a chart of all the min and max values for each of the scan files:
Depending on the config you entered, you will have all the spectrograms already pushed to the AWS cloud in S3 storage or just locally on your R-PI microSD.
You can use a simple FTP for Windows program to download them to your laptop. An example one is WinSCP .
If you want to give more scientific value to your data, you need to calibrate the receiver output so that it can be expressed in units of measure like db. I'm leaving this procedure to a future improvement if I will see good output and acceptable RFI to manage on my site.
If you enabled the upload to S3 with the option uploadToS3 = True, the output spectrograms are browsable online at this URLs:
You can check your ability to send mqtt notifications using the 2 programs awsiotsub.py and awsiotpub.py (in this order): I kindly ask you get in touch with me if interested to get the IoT certificate files enabling this feature.
All this effort could not have been put in place without the love and support of my beloved wife Chiara.
Hope you enjoy experimenting with this stuff like I did and I'm doing. Will certainly update this or extend with my next results, improvements. There are several things that can be changed or simplified and I'm happy to receive your feedback/suggestions/corrections...