Merge pull request #7 from qoherent/5-transition-away-from-conda

Transition away from Conda

Author: mrl280

.gitignore

@@ -22,6 +22,8 @@ venv/
 # Visual Studio Code specific file
 .vscode/
 
-# Project Related Files
-/SpectrumSensingDataset/
+# Lightning
 /lightning_logs/
+
+# Project dataset
+/spectrum_sensing_dataset.hdf5

README.md

@@ -1,19 +1,19 @@
 # Spectrogram Segmentation
 
-The successful application of [semantic segmentation](https://www.ibm.com/topics/semantic-segmentation) to radiofrequency (RF) spectrograms holds significant applications 
-for [spectrum sensing](https://iopscience.iop.org/article/10.1088/1742-6596/2261/1/012016#:~:text=In%20cognitive%20radio%2C%20spectrum%20sensing,user%20can%20use%20the%20spectrum.) and serves as a foundational example showcasing the near-term feasibility of 
+The successful application of [semantic segmentation](https://www.ibm.com/topics/semantic-segmentation) to radiofrequency (RF) spectrograms has significant 
+implications for [spectrum sensing](https://iopscience.iop.org/article/10.1088/1742-6596/2261/1/012016#:~:text=In%20cognitive%20radio%2C%20spectrum%20sensing,user%20can%20use%20the%20spectrum.), and serves as a foundational example showcasing the near-term feasibility of 
 [intelligent radio](https://www.qoherent.ai/intelligentradio/) technology.
 
 In this example, we use [PyTorch](https://pytorch.org/) and [Lightning](https://lightning.ai/docs/pytorch/stable/) to train a segmentation model to identify and
 differentiate between 5G NR and 4G LTE signals within wideband spectrograms.
 
 Qoherent's mission to drive the creation of intelligent radio technology requires a combination of open-source and 
 proprietary tools. This example, which leverages open-source tools and machine learning frameworks to train on 
-synthetic radio data generated using MATLAB, showcases our commitment to interoperability and our tool-agnostic 
-approach to innovation.
+synthetic radio data generated using MATLAB's powerful 5G and LTE toolboxes, showcases our commitment to
+interoperability and our tool-agnostic approach to innovation.
 
-Classification results are comparable to those reported by MathWorks' AI-based network. For more information, 
-please refer to the following article by MathWorks:
+Classification results are comparable to those achieved by MathWorks' custom network, albeit with more learnables. 
+For more information, please refer to the following article by MathWorks: 
 [Spectrum Sensing with Deep Learning to Identify 5G and LTE Signals](https://www.mathworks.com/help/comm/ug/spectrum-sensing-with-deep-learning-to-identify-5g-and-lte-signals.html).
 
 If you found this example interesting or helpful, don't forget to give it a star! ⭐
@@ -24,15 +24,15 @@ If you found this example interesting or helpful, don't forget to give it a star
 This example is provided as a Jupyter Notebook. You have the option to either run this example locally or in Google 
 Colab.
 
-To run this example locally, you'll need to download the project and dataset and set up a Conda 
-virtual environment. If this seems daunting, we recommend running this example on Google Colab.
+To run this example locally, you'll need to download the project and dataset and set up a Python 
+virtual environment. If this seems daunting, we recommend running this example on Google Colab (Coming soon!).
 
 ### Running this example locally
 
-Please note that running this example locally will require approximately 10 GB of free space. Please ensure you 
+Please note that running this example locally will require approximately 6.1 GB of free space. Please ensure you 
 have sufficient space available prior to proceeding.
 
-1. Ensure that [Git](https://git-scm.com/downloads) and [Conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) are installed on the computer where you plan to run this example. 
+1. Ensure that [Git](https://git-scm.com/downloads) and [Python](https://www.python.org/downloads/) are installed on the computer where you plan to run this example. 
 Additionally, if you'd like to accelerate model training with a GPU, you'll require [CUDA](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html).
 
 
@@ -42,58 +42,94 @@ git clone https://github.com/qoherent/spectrogram-segmentation.git
 ```
 
 
-3. Create a Conda environment using the provided `environment.yml` file:
+3. Create and activate a Python [virtual environment](https://docs.python.org/3/library/venv.html). This is a best practice for isolating project dependencies.
+
+<details>
+<summary><strong>Windows</strong></summary>
+
+Use the following command to create a new directory named `venv` within the project directory:
+```commandline
+python -m venv venv
+```
+
+Then, activate the virtual environment with:
+```commandline
+venv\Scripts\activate
+```
+
+</details>
+
+<details>
+<summary><strong>Linux/Mac</strong></summary>
+
+Use the following command to create a new directory named `venv` within the project directory:
 ```commandline
-conda env create -f environment.yml
+python3 -m venv venv
 ```
-This will create a new Conda environment named `spectrogram-segmentation` within the Conda installation directory.
 
+Then, activate the virtual environment with:
+```commandline
+source venv/bin/activate
+```
+
+</details>
+
+Activating the virtual environment should modify the command prompt to show `(venv)` at the beginning, indicating 
+that the virtual environment is active.
 
-4. Active the environment:
+
+4. Install project dependencies from the provided `requirements.txt` file:
 ```commandline
-conda activate spectrogram-segmentation
+pip install -r requirements.txt
 ```
 
 
-5. Download and unpack the spectrum sensing dataset:
+5. Download the spectrum sensing dataset.
+
+<details>
+<summary><strong>Windows</strong></summary>
+
 ```commandline
 python download_dataset.py
 ```
-This command will create a new directory named `SpectrumSensingDataset` at the project's root. The 
-MathWorks Spectrum Sensing dataset will be downloaded and unpacked into this directory automatically.
+
+</details>
+
+<details>
+<summary><strong>Linux/Mac</strong></summary>
+
+```commandline
+python3 download_dataset.py
+```
+
+</details>
+
+This will download the `spectrum_sensing_dataset.hdf5` source file to the project's root directory.
 
 
 6. Register the environment kernel with Jupyter:
 ```commandline
-ipython kernel install --user --name=spectrogram-segmentation
+ipython kernel install --user --name=venv --display-name "Spectrogram Segmentation (venv)"
 ```
 
 
-7. Open the notebook, `spectrogram_segmentation.ipynb`, specifying to use the `spectrogram-segmentation` kernel:
+7. Open the notebook, `spectrogram_segmentation.ipynb`, specifying to use the new kernel:
 ```commandline
-jupyter notebook spectrogram_segmentation.ipynb --MultiKernelManager.default_kernel_name=spectrogram-segmentation
+jupyter notebook spectrogram_segmentation.ipynb --MultiKernelManager.default_kernel_name=venv
 ```
 
 
 8. Give yourself a pat on the back - you're all set up and ready to explore the example! For more information on 
-navigating the Jupyter Notebook interface and executing code, please check out this tutorial by the Codecademy 
+navigating the Jupyter Notebook interface and executing code cells, please check out this tutorial by the Codecademy 
 Team: [How To Use Jupyter Notebooks](https://www.codecademy.com/article/how-to-use-jupyter-notebooks).
 
 Depending on your system specifications and the availability of a CUDA, running this example locally may take 
 several minutes. If a cell is taking too long to execute, you can interrupt its execution by clicking the "Kernel" 
 menu and selecting "Interrupt Kernel" or by pressing `Ctrl + C` in the terminal where Jupyter Notebook is running.
 
 
-9. After you finish exploring, consider removing the dataset from your system and deleting the Conda environment to 
-free up space. You can delete the Conda environment using the following command:
-```commandline
-conda env remove --name spectrogram-segmentation
-```
-
-### Running this example in Google Colab
-
-**Coming soon:** Don't want the hassle of downloading the project and dataset and setting up a Conda environment? 
-We've shared the notebook on Google Colab: [Spectrogram Segmentation]().
+9. After you finish exploring, consider removing the dataset from your system and deleting the virtual environment to 
+free up space. Remember to deactivate the virtual environment using the deactivate command before deleting the folder.
 
 
 ## 🤝 Contribution
@@ -119,10 +155,10 @@ This work is a product of the collaborative efforts of the Qoherent team. Of spe
 
 ## 🙏 Attribution
 
-The dataset used in this example was prepared by MathWorks and is publicly available [here](https://www.mathworks.com/supportfiles/spc/SpectrumSensing/SpectrumSenseTrainingDataNetwork.tar.gz). For more information 
-on how this dataset was generated or to generate further spectrum data, please refer to MathWork's article on spectrum 
-sensing. For more information about Qoherent's use of MATLAB to accelerate intelligent radio research, check out our 
-[customer story](https://www.mathworks.com/company/user_stories/qoherent-uses-matlab-to-accelerate-research-on-next-generation-ai-for-wireless.html).
+The dataset used in this example was prepared by MathWorks using their 5G and LTE toolboxes and is publicly available
+[here](https://www.mathworks.com/supportfiles/spc/SpectrumSensing/SpectrumSenseTrainingDataNetwork.tar.gz). For more information on how this dataset was generated or to generate further spectrum data, please refer
+to MathWork's article on spectrum sensing. For more information about Qoherent's use of MATLAB to accelerate 
+intelligent radio research, check out our [customer story](https://www.mathworks.com/company/user_stories/qoherent-uses-matlab-to-accelerate-research-on-next-generation-ai-for-wireless.html).
 
 The DeepLabv3 models used in this example were initially proposed by Chen _et al._ and are further discussed 
 in their 2017 paper titled '[Rethinking Atrous Convolution for Semantic Image Segmentation](https://arxiv.org/abs/1706.05587)'. The MobileNetV3 

download_dataset.py

@@ -2,46 +2,58 @@
 Download MathWorks' Spectrum Sensing dataset, if it isn't already downloaded.
 """
 
+import hashlib
 import os
-import tarfile
-import warnings
 
 import requests
 from torch.utils.model_zoo import tqdm
 
-mirror = "https://www.mathworks.com/supportfiles/spc/SpectrumSensing/"
-resource = "SpectrumSenseTrainingDataNetwork.tar.gz"
+
+def sha256(target: str) -> str:
+    """Calculates the SHA256 hash of the target resource.
+
+    :param target: The full path, including the filename, of the resource to hash.
+    :type target: str
+
+    :return: The SHA256 hash of the target resource.
+    """
+    sha256_hash = hashlib.sha256()
+
+    with open(target, "rb") as file:
+        for chunk in iter(lambda: file.read(4096), b""):
+            sha256_hash.update(chunk)
+
+    return sha256_hash.hexdigest()
+
+
+mirror = "https://storage.googleapis.com/qoherent_external_drive/general_dataset_library/"
+resource = "spectrum_sensing_dataset_v1.0.hdf5"
 file_url = "{}{}".format(mirror, resource)
-target_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "SpectrumSensingDataset")
-archive_file = os.path.join(target_dir, resource)
+sha256_checksum = "8a93aa14145ea1a35cbc191defbbcf90c49ecdb89e6e93f3e55357f182d184c6"
 
-# Ensure the target directory exists.
-if not os.path.exists(target_dir):
-    os.makedirs(target_dir, exist_ok=True)
+target = os.path.join(os.path.dirname(os.path.abspath(__file__)), "spectrum_sensing_dataset.hdf5")
 
-# Check if the target directory is empty, and abort the download if it's not.
-if len(os.listdir(target_dir)) != 0:
-    warnings.warn(f"Target directory '{target_dir}' is not empty. Aborting download.")
+# Check if the dataset source file already exists.
+if os.path.exists(target):
+    print(f"{target} already exists. Aborting download.")
     exit(1)
 
-# Download and extract the dataset into the target directory.
+# Download the dataset source file into the target directory.
 print(f"Downloading {format(file_url)}")
 n_bytes = int(requests.head(file_url).headers.get("Content-Length", 0))
 
 with (
     requests.get(file_url, stream=True, timeout=3) as r,
-    open(archive_file, "wb") as out_file,
-    tqdm(desc="Downloading", total=n_bytes, unit="B", unit_scale=True) as pbar,
+    open(target, "wb") as out_file,
+    tqdm(desc="Downloading MathWorks' Spectrum Sensing Dataset", total=n_bytes, unit="B", unit_scale=True) as pbar,
 ):
     for chunk in r.iter_content(chunk_size=1024):
         if chunk:
             out_file.write(chunk)
             pbar.update(len(chunk))
 
-print(f"Extracting to {target_dir}")
-with tarfile.open(archive_file) as tar:
-    tar.extractall(target_dir)
-
-# Remove the archive file after extraction.
-os.remove(archive_file)
-print(f"Archive file '{archive_file}' removed.")
+if sha256_checksum != sha256(target=target):
+    raise RuntimeError(
+        f"Checksum of {target} does not match expected.\n"
+        f"The download may be corrupted, please remove the corrupted resource and try again."
+    )

environment.yml

@@ -0,0 +1,17 @@
+--extra-index-url https://download.pytorch.org/whl/cu121  # CUDA 12.1 support for torch and torchvision.
+# --extra-index-url https://download.pytorch.org/whl/cu118  # CUDA 11.8 support for torch and torchvision.
+
+h5py==3.11.0
+jupyter==1.0.0  # Jupyter system, including the notebook and the IPython kernel, all in one go.
+lightning==2.3.3
+matplotlib==3.9.1
+numpy==1.26.4
+pandas==2.2.2
+pillow==10.4.0
+requests==2.32.3
+scikit-learn==1.5.1
+scipy==1.14.0
+tabulate==0.9.0
+torch==2.3.1
+torchmetrics==1.4.0
+torchvision==0.18.1

requirements.txt

@@ -0,0 +1,4 @@
+black[jupyter]==24.4.2
+flake8==7.1.0
+flake8-nb==0.5.3
+isort==5.13.2