retrain HSCNN+ on airborne RGB and HSI data

[MarconiS]

Hi,
I have a dataset of RGB and HSI data co-registered from an airborne platform, with RGB resolution at 10cm and hsi at 1m. I would love to try HSCNN+ to reconstruct the HSI signature and see if that can be used to increase the HSI data resolution.
Would it be possible by just changing the training data?
How should I prepare/format the data to fit the structure of HSCNN+?

Thanks,
Sergio

[marc-micatka]

This is quite old but I'll add my input here:

If you look at the data_loader.py file inside "hscnn-d_real/ (or clean)" you'll see this:

filename_queue = tf.compat.v1.train.string_input_producer(
    [filename], num_epochs=num_epoch)
reader = tf.compat.v1.TFRecordReader()

_, serialized_example = reader.read(filename_queue)
features = tf.io.parse_single_example(
    serialized=serialized_example,
    features={'data': tf.io.FixedLenFeature([87500], tf.float32)})

The data_loader is expecting a .tfrecords file with your HSI and RGB images stacked on top of each other in a float32 field with the name 'data'.

You can generate this data in this fashion:

writer = tf.io.TFRecordWriter(tf_records_filename)
 data_path = '/path/to/directory/'

for index in range(len(data_path)):
    hs_img = load_img(hs_file_list[index])
    rgb_img = load_img(rgb_file_list[index])

    img_np = tf.concat([hs_img, rgb_img], axis=2).numpy()
    normalized_data = (img_np - np.min(img_np)) / np.ptp(img_np)

    example = tf.train.Example(features=tf.train.Features(
                feature={'data': _floats_feature(normalized_data)}))
    writer.write(example.SerializeToString())

And _floats_feature is defined:

def _floats_feature(value):
    return tf.train.Feature(float_list=tf.train.FloatList(
        value=value.reshape(-1)))

This creates a TFRecords() file (.tfrecords) with a flattened numpy array as your input. You can see in the data_loader file that the array is reshaped into size 50x50xbands and 50x50x3 from an overall size of 50x50x34 (34 = 31 bands + 3 channels). This is why the data_loader does this:

features={'data': tf.io.FixedLenFeature([85000], tf.float32)})

85000 =50 x 50 x34

Once saved, you should be able to pass this filename into the dataloader as written. You should generate both a training dataset and a validation dataset (check the train.py file for flags and to change the path to reference your data location).

Bear in mind that the paper uses 50x50 pixel patches for training so you should probably do the same.
You can do this via tf.image.random_crop() or tf.image.get_patches().