Understanding HOMO/LUMO Data

Open In Colab

In the previous notebook, you explored the dataset HOPV_stability.csv, which could be read directly into a pandas DataFrame. In this notebook, you will work with HOPV_homolumo.data. Unlike the CSV file, this dataset requires additional processing before it can be used in analysis.

HOPV_homolumo.data is a text file containing information on 350 molecules, the same set of molecules as in HOPV_stability.csv. While the file contains many variables, your main focus will be on:

The file also includes additional data. Extract these variables as well, since they may provide useful insights for further exploration.

Step 1: Parse the file and organize the data into the expected tabular format, as illustrated below:

  smiles doi InChIKEY construction architecture complement homo lumo electrochemical_gap optical_gap pce voc jsc ff
0 Cc1ccc(-c2c3cc(-c4ccc(-c5sc(-c6cccs6)c6c… 10.1021/ar5000743 BOUZSMSNJFUUCG-UHFFFAOYSA-N polymer bulk PC71BM -5.40 -3.54 1.86 1.59 7.81 0.69 17.07 66.3
1 CN1c2ccccc2C(=C2c3ccc(-c4cccs4)cc3N(C)C2… 10.1039/C3PY01598J PAPOKCOIPOLVSC-BNEIJSFPSA-N polymer bulk PC71BM nan nan nan nan 3.0 0.89 5.4 63.0
2 Cc1cc(-c2cc3c4nsnc4c(-c4cc(C)c(-c5cccs5)… 10.1021/ja210687r IKPPCQWOLGDYAQ-UHFFFAOYSA-N polymer bulk PC61BM -5.45 -3.81 1.64 1.54 6.3 0.76 12.0 69.0
3 Cn1c2ccccc2c2ccc(-c3ccc(-c4ccc(C5=C(c6cc… 10.1021/ar900061z LZAUYQACJPMHMN-UHFFFAOYSA-N polymer bulk PC61BM nan nan nan 2.0 1.51 0.75 4.68 44.5
4 Cc1cc(-c2c3cc(-c4cccs4)sc3c(-c3cc(C)c(C)… 10.1039/C3PY01598J RJIQMHRLDYGXFS-HYXXFLLTSA-N polymer bulk PC71BM -5.65 -3.97 1.68 nan 5.7 0.9 10.16 62.0
… … … … … … … … … … … … … … …
345 CC1(C)c2ccccc2-c2ccc(-c3ccc(C4=C(c5ccccc… 10.1021/ar900061z KFVMXISUDWBOQO-UHFFFAOYSA-N polymer bulk PC61BM -5.19 -3.38 1.81 2.08 2.01 0.65 8.67 35.8
346 Cc1ccc(-c2c3ccsc3c(-c3ccc(C)s3)c3cc(-c4c… 10.1021/ar5000743 FIQSDHZNBUEQOK-UHFFFAOYSA-N polymer bulk PC71BM -5.57 -3.79 1.78 1.445 6.6 0.73 14.0 65.0
347 C[Si]1(C)c2ccccc2-c2ccc(-c3ccc(-c4ccc(-c… 10.1039/b9py00325h KSAPXMCRKXFLEP-UHFFFAOYSA-N polymer bulk PC61BM -5.44 nan nan 1.8 5.4 0.9 9.5 51.0
348 N#CC(=Cc1ccc(C=Cc2ccc(-c3ccc(N(c4ccccc4)… 10.1039/B908802D ZVGFLVHIOZGBRL-SGVMFCADSA-N molecule DSSC TiO2 nan nan nan nan 2.73 0.635 6.55 66.0
349 C=Cc1cc(OC)c(C=Cc2ccc(N(c3ccccc3)c3ccc(C… 10.1021/ma100943t BXWZTSDIKYVFBE-OGVDJYGPSA-N polymer bulk PC61BM -5.30 -3.49 1.81 1.68 4.06 0.87 8.35 56.0

350 rows × 14 columns

Expected output format.

Step 2: Perform the same exploratory checks and analyses you applied to HOPV_stability.csv, this time on the data from HOPV_homolumo.data.

Working in Colab? This notebook uses rdkit, which Colab does not include. Run the cell below: it installs the package only when it detects Colab. Locally it does nothing, because the course environment already includes rdkit.

# Installs the missing packages on Google Colab only; does nothing locally.
import sys

if "google.colab" in sys.modules:
    %pip install -q rdkit
# IMPORT WHAT YOU NEED HERE

Your Turn

Read HOPV_homolumo.data and print the first 10 lines. This lets us take a first look at the raw text and get a sense of the file’s structure before we start parsing it.

Working in Colab? The repository’s data/ folder is not available there. Download the file into your Colab session first, then use "HOPV_homolumo.data" as the path: !wget -q https://raw.githubusercontent.com/BredaUniversityADSAI/introduction-to-ml/main/data/001_raw_data/HOPV_homolumo.data

# YOUR CODE HERE

This time, each row does not simply correspond to a SMILES string and its targets.

Your Turn

How many lines are there? Expected answer is 374852.

# YOUR CODE HERE

From the Data Understanding page of Iteration 1, we know that this file contains data on 350 molecules. This means:

num_molecules = 350
print(f"On average, there are {len(lines)/num_molecules:.3f} lines per molecule")

To determine how to extract the values for each molecule, you will need to consult the paper.

Step 1: Parse the file and organize the data into the expected tabular format

Your Turn

Take a look at Table 1 in the HOPV data descriptor (Lopez et al., 2016, open access) and answer the following:

  1. Which line contains the SMILES string?
  2. Which line contains the experimental data?

Keep in mind the fact that Python is zero indexed.

Your Turn

Now extract the corresponding lines. Expected output:

SMILES should be:

'Cc1ccc(-c2c3cc(-c4ccc(-c5sc(-c6cccs6)c6cc(S(C)(=O)=O)sc65)s4)sc3c(-c3ccc(C)s3)c3ccsc32)s1\n'

experimental values should be:

'10.1021/ar5000743,BOUZSMSNJFUUCG-UHFFFAOYSA-N,polymer,bulk,PC71BM,-5.40,-3.54,1.86,1.59,7.81,0.69,17.07,66.3\n'

# YOUR CODE HERE
smiles_line = None
# YOUR CODE HERE
experimental_line = None

Notice the \n character at the end of the lines. When working with text data, always account for newline characters (\n) by handling or removing them as needed.

Your Turn

Table 2 in the data descriptor document describes the CSV format for storing experimental information (experimental_line above).

Write a function called experimental_vals_to_dict to parse the experimental line and return a dictionary where each key is from Table 2 and corresponding experimental values are from the line itself.

Call the function with the experimental line you extracted previously.

output_dictionary = experimental_vals_to_dict(table2_keys, experimental_line)

The expected output is:

{'doi': '10.1021/ar5000743',
 'InChIKEY': 'BOUZSMSNJFUUCG-UHFFFAOYSA-N',
 'construction': 'polymer',
 'architecture': 'bulk',
 'complement': 'PC71BM',
 'homo': '-5.40',
 'lumo': '-3.54',
 'electrochemical_gap': '1.86',
 'optical_gap': '1.59',
 'pce': '7.81',
 'voc': '0.69',
 'jsc': '17.07',
 'ff': '66.3'}

Note that the numbers are kept as strings, we will handle those later.

table2_keys = [
    'doi',
    'InChIKEY',
    'construction',
    'architecture',
    'complement',
    'homo',
    'lumo',
    'electrochemical_gap',
    'optical_gap',
    'pce',
    'voc',
    'jsc',
    'ff'
]
def experimental_vals_to_dict(table2_keys, experimental_line):
    """
    Convert a single line of experimental data into a dictionary.

    Each line of data is a comma-separated string (CSV format) that contains
    values such as doi, InChIKEY, molecular properties, and experimental results.
    The provided list of keys is matched to these values to create a dictionary.

    Parameters
    ----------
    table2_keys : list of str
        The list of column names (keys) to assign to each value.
    experimental_line : str
        A string containing one line of comma-separated values.

    Returns
    -------
    dict
        A dictionary where the keys come from `table2_keys` and the values
        come from `experimental_line`.
    """
    # YOUR CODE HERE
    d = None
    return d
output_dictionary = experimental_vals_to_dict(table2_keys, experimental_line)
output_dictionary

Your Turn

We have all the information we need from the first molecule in the file, the SMILES string and the experimental values containing HOMO and LUMO. Let’s put them together into a dataframe of single row.

  smiles doi InChIKEY construction architecture complement homo lumo electrochemical_gap optical_gap pce voc jsc ff
0 Cc1ccc(-c2c3cc(-c4ccc(-c5sc(-c6cccs6)c6c… 10.1021/ar5000743 BOUZSMSNJFUUCG-UHFFFAOYSA-N polymer bulk PC71BM -5.40 -3.54 1.86 1.59 7.81 0.69 17.07 66.3

1 rows × 14 columns

Expected output.

# YOUR CODE HERE

Your Turn

Display the molecule in the notebook based on the SMILES string using RDKit. You can use the function MolFromSmiles for this. It converts a SMILES string into a molecule object that Jupyter can display directly.

from rdkit.Chem import MolFromSmiles

mol = MolFromSmiles(# put the string here)
mol  # Jupyter will automatically display the molecule

documentation:

https://www.rdkit.org/docs/source/rdkit.Chem.rdmolfiles.html#rdkit.Chem.rdmolfiles.MolFromSmiles

Molecule drawn by RDKit
Expected output.
# YOUR CODE HERE

Your Turn

Extend your approach to all 350 molecules in the file. We estimated earlier that there are about 1071 lines per molecule, but this is only an average, some molecules have fewer lines, others more. Therefore, you cannot simply slice the file like this:

molecule_1 = line[0:1071]
molecule_2 = line[1071:1071*2]
...

Instead, you must identify a repeated structure in the file that marks where one molecule ends and the next begins.

Notice that the file does not contain simple labels like Molecule 1, Molecule 2, etc. It’s up to you to explore the text and discover a consistent pattern that can be used to determine the boundaries of each molecule. Your goal is not to extract everything, but at minimum to locate the SMILES line and the experimental line for each molecule.

Open the text file and scroll through it carefully. Look for repeated markers that indicate the start of a new molecule. Once you have a candidate pattern, test it by counting how many times it appears. If your count is 350, you’ve found the right boundary marker.

counter = 0
for line in lines:
    # YOUR CODE HERE
    # write some logic to count the molecules
    # counter should print 350
    counter += 1
print(counter)

Your Turn

Using the logic above write a function called hopv_parser to find all the molecules and extract the SMILES and the experimental line, store it into a dataframe. Expected output:

  smiles doi InChIKEY construction architecture complement homo lumo electrochemical_gap optical_gap pce voc jsc ff
0 Cc1ccc(-c2c3cc(-c4ccc(-c5sc(-c6cccs6)c6c… 10.1021/ar5000743 BOUZSMSNJFUUCG-UHFFFAOYSA-N polymer bulk PC71BM -5.40 -3.54 1.86 1.59 7.81 0.69 17.07 66.3
1 CN1c2ccccc2C(=C2c3ccc(-c4cccs4)cc3N(C)C2… 10.1039/C3PY01598J PAPOKCOIPOLVSC-BNEIJSFPSA-N polymer bulk PC71BM nan nan nan nan 3.0 0.89 5.4 63.0
2 Cc1cc(-c2cc3c4nsnc4c(-c4cc(C)c(-c5cccs5)… 10.1021/ja210687r IKPPCQWOLGDYAQ-UHFFFAOYSA-N polymer bulk PC61BM -5.45 -3.81 1.64 1.54 6.3 0.76 12.0 69.0
3 Cn1c2ccccc2c2ccc(-c3ccc(-c4ccc(C5=C(c6cc… 10.1021/ar900061z LZAUYQACJPMHMN-UHFFFAOYSA-N polymer bulk PC61BM nan nan nan 2.0 1.51 0.75 4.68 44.5
4 Cc1cc(-c2c3cc(-c4cccs4)sc3c(-c3cc(C)c(C)… 10.1039/C3PY01598J RJIQMHRLDYGXFS-HYXXFLLTSA-N polymer bulk PC71BM -5.65 -3.97 1.68 nan 5.7 0.9 10.16 62.0
… … … … … … … … … … … … … … …
345 CC1(C)c2ccccc2-c2ccc(-c3ccc(C4=C(c5ccccc… 10.1021/ar900061z KFVMXISUDWBOQO-UHFFFAOYSA-N polymer bulk PC61BM -5.19 -3.38 1.81 2.08 2.01 0.65 8.67 35.8
346 Cc1ccc(-c2c3ccsc3c(-c3ccc(C)s3)c3cc(-c4c… 10.1021/ar5000743 FIQSDHZNBUEQOK-UHFFFAOYSA-N polymer bulk PC71BM -5.57 -3.79 1.78 1.445 6.6 0.73 14.0 65.0
347 C[Si]1(C)c2ccccc2-c2ccc(-c3ccc(-c4ccc(-c… 10.1039/b9py00325h KSAPXMCRKXFLEP-UHFFFAOYSA-N polymer bulk PC61BM -5.44 nan nan 1.8 5.4 0.9 9.5 51.0
348 N#CC(=Cc1ccc(C=Cc2ccc(-c3ccc(N(c4ccccc4)… 10.1039/B908802D ZVGFLVHIOZGBRL-SGVMFCADSA-N molecule DSSC TiO2 nan nan nan nan 2.73 0.635 6.55 66.0
349 C=Cc1cc(OC)c(C=Cc2ccc(N(c3ccccc3)c3ccc(C… 10.1021/ma100943t BXWZTSDIKYVFBE-OGVDJYGPSA-N polymer bulk PC61BM -5.30 -3.49 1.81 1.68 4.06 0.87 8.35 56.0

350 rows × 14 columns

Expected output.

def hopv_parser(lines):
    """
    EXPLAIN HOW IT WORKS HERE

    Parameters
    ----------
    lines : list of str
        Lines read from a HOPV_homolumo.data file.

    Returns
    -------
    pandas.DataFrame
        DataFrame where each row corresponds to a molecule.

    """
    # YOUR CODE HERE
    df = None
    return df
df = hopv_parser(lines)
df

Now we are ready for data understanding. However, before doing so, save the dataframe as a csv file so that the next we would like to use this dataset we do not have to parse HOPV_homolumo.data.

Save the DataFrame as HOPV_homolumo_processed.csv into the folder 002_processed_data.

# YOUR CODE HERE

Step 2: Data Understanding

Your Turn

Great job! You’ve successfully converted the raw data into a table in the desired format. Now we can investigate the dataframe.

Let’s first look at handling empty values. Run the following code df.isna().any(). It returns a Boolean Series where each column is marked True if it contains at least one NaN value, and False otherwise. Answer the following:

Which columns contain empty values?

# YOUR CODE

It seems there are no NaN values, yet the DataFrame summary clearly shows missing values. Why does this happen?

  smiles doi InChIKEY construction architecture complement homo lumo electrochemical_gap optical_gap pce voc jsc ff
0 Cc1ccc(-c2c3cc(-c4ccc(-c5sc(-c6cccs6)c6c… 10.1021/ar5000743 BOUZSMSNJFUUCG-UHFFFAOYSA-N polymer bulk PC71BM -5.40 -3.54 1.86 1.59 7.81 0.69 17.07 66.3
1 CN1c2ccccc2C(=C2c3ccc(-c4cccs4)cc3N(C)C2… 10.1039/C3PY01598J PAPOKCOIPOLVSC-BNEIJSFPSA-N polymer bulk PC71BM nan nan nan nan 3.0 0.89 5.4 63.0
2 Cc1cc(-c2cc3c4nsnc4c(-c4cc(C)c(-c5cccs5)… 10.1021/ja210687r IKPPCQWOLGDYAQ-UHFFFAOYSA-N polymer bulk PC61BM -5.45 -3.81 1.64 1.54 6.3 0.76 12.0 69.0
3 Cn1c2ccccc2c2ccc(-c3ccc(-c4ccc(C5=C(c6cc… 10.1021/ar900061z LZAUYQACJPMHMN-UHFFFAOYSA-N polymer bulk PC61BM nan nan nan 2.0 1.51 0.75 4.68 44.5
4 Cc1cc(-c2c3cc(-c4cccs4)sc3c(-c3cc(C)c(C)… 10.1039/C3PY01598J RJIQMHRLDYGXFS-HYXXFLLTSA-N polymer bulk PC71BM -5.65 -3.97 1.68 nan 5.7 0.9 10.16 62.0
… … … … … … … … … … … … … … …
345 CC1(C)c2ccccc2-c2ccc(-c3ccc(C4=C(c5ccccc… 10.1021/ar900061z KFVMXISUDWBOQO-UHFFFAOYSA-N polymer bulk PC61BM -5.19 -3.38 1.81 2.08 2.01 0.65 8.67 35.8
346 Cc1ccc(-c2c3ccsc3c(-c3ccc(C)s3)c3cc(-c4c… 10.1021/ar5000743 FIQSDHZNBUEQOK-UHFFFAOYSA-N polymer bulk PC71BM -5.57 -3.79 1.78 1.445 6.6 0.73 14.0 65.0
347 C[Si]1(C)c2ccccc2-c2ccc(-c3ccc(-c4ccc(-c… 10.1039/b9py00325h KSAPXMCRKXFLEP-UHFFFAOYSA-N polymer bulk PC61BM -5.44 nan nan 1.8 5.4 0.9 9.5 51.0
348 N#CC(=Cc1ccc(C=Cc2ccc(-c3ccc(N(c4ccccc4)… 10.1039/B908802D ZVGFLVHIOZGBRL-SGVMFCADSA-N molecule DSSC TiO2 nan nan nan nan 2.73 0.635 6.55 66.0
349 C=Cc1cc(OC)c(C=Cc2ccc(N(c3ccccc3)c3ccc(C… 10.1021/ma100943t BXWZTSDIKYVFBE-OGVDJYGPSA-N polymer bulk PC61BM -5.30 -3.49 1.81 1.68 4.06 0.87 8.35 56.0

350 rows × 14 columns

nan values are visible on homo, lumo and the gap columns.

To understand why, let’s take a single nan value and check its type:

df.iloc[1]['homo']
type(df.iloc[1]['homo'])

As you can see, the issue is that the text file encodes missing values as the string "nan" rather than as actual NaN values. As a result, Python and pandas interpret them as regular non-empty strings. To handle them correctly as missing data, we need to replace these strings with np.nan.

pd.isna('nan') # 'nan' is not really nan
pd.isna(np.nan) # np.nan is what we want

Your Turn

Replace all "nan" values with np.nan and run the code df.isna().any() to identify columns that contain empty values

# YOUR CODE HERE

As you can see, almost every column contains empty values except for the smiles column. The key takeaway is that you must approach your data like a detective. Just because df.isna().any() didn’t flag any empty columns doesn’t mean there aren’t missing values, even if your code is technically correct. You will face many such situations in data science and machine learning. The only way to handle them properly is by knowing the fundamentals and by staying skeptical.

Your Turn

Calculate the number of rows that fall into each category:
- Both HOMO and LUMO are present (expected 233) - HOMO is present and LUMO is missing (expected 12) - HOMO is missing and LUMO is present (expected 0) - Both HOMO and LUMO are missing (expected 105)

# YOUR CODE HERE

Your Turn

Next, check for duplicates in the SMILES column. Why do we care about duplicates?

# YOUR CODE HERE

Your Turn

Remember when we were parsing the experimental line, we left the numbers as strings. It is time to convert them to floats, so that we can calculate some descriptive statistics on them and plot their distribution.

  1. Check the data type of each column
  2. Identify which columns should be converted to numerical data types
  3. Identify if these are to be convert to a float or int
  4. Do the conversion
  5. Convert the remaining columns to string
  6. Check the data type of each column to verify conversion

Note: In pandas, text data often shows up with the dtype object, because pandas is built on top of NumPy, which was originally designed for numbers, not text. NumPy didn’t have a special type for variable-length strings, so it stored them as generic Python objects, and pandas inherited this behavior. This means that when you see object, it usually means “a column of Python strings.” More recently, pandas introduced a dedicated string dtype, which makes working with text clearer and more consistent. In practice, both work, but using string is preferred in new projects since it avoids some problems with object columns and handles missing values better.

Expected output:

smiles                 string[python]
doi                    string[python]
InChIKEY               string[python]
construction           string[python]
architecture           string[python]
complement             string[python]
homo                          float64
lumo                          float64
electrochemical_gap           float64
optical_gap                   float64
pce                           float64
voc                           float64
jsc                           float64
ff                            float64
# YOUR CODE HERE

Now that we have parsed the text file, cleaned out empty values, and converted the columns to the correct data types, we can take a closer look at the dataset. While the data descriptor document specifies what each field represents, the meanings may not be immediately clear without a chemistry background. Below is a plain-language explanation of each column, prepared with the help of ChatGPT.

  1. Digital Object Identifier (DOI)

    • A permanent identifier of the research paper where the experiment was published.
    • Useful for tracing back to the original experimental setup and conditions.
  2. InChIKEY of molecule

    • A hashed version of the InChI (International Chemical Identifier) string.
    • Provides a unique, compact ID for the molecule.
    • Unlike SMILES, it is not used directly for feature extraction, but it ensures uniqueness and allows linking to chemical databases.
  3. Construction (Polymer / Molecule)

    • Whether the active layer material is a single small molecule or a polymer.
    • This affects how molecules are processed experimentally, and may correlate with properties like stability or efficiency.
  4. Architecture

    • The type of device structure used in the solar cell (e.g., bulk heterojunction).
    • Relevant for interpreting performance metrics, but not directly a molecular property.
  5. Complement

    • The electron acceptor or donor material paired with the molecule in the solar cell.
    • For example, PCBM derivatives are often listed here.
    • This is important for device-level performance
  6. HOMO (Highest Occupied Molecular Orbital)

    • The energy level of the highest filled electron orbital (in eV).
    • A molecular property that influences how easily the molecule donates electrons.
    • Prediction target for your ML task.
  7. LUMO (Lowest Unoccupied Molecular Orbital)

    • The energy level of the lowest empty electron orbital (in eV).
    • Indicates how easily the molecule can accept electrons.
    • Prediction target for your ML task.
  8. Electrochemical gap

    • The difference between LUMO and HOMO (LUMO − HOMO).
    • Can be measured experimentally, often with cyclic voltammetry.
    • Correlates with the open-circuit voltage of a device.
  9. Optical gap

    • The band gap inferred from absorption spectra (UV-Vis).
    • Related but not identical to the HOMO-LUMO gap because of excitonic effects.
  10. PCE (Power Conversion Efficiency)

    • Percentage of sunlight converted into electricity by the device.
    • A device-level metric, depends not only on the molecule but also the architecture and complement.
  11. VOC (Open-Circuit Voltage)

    • The maximum voltage a device can deliver when no current flows.
    • Influenced by HOMO of donor and LUMO of acceptor.
  12. JSC (Short-Circuit Current Density)

    • Current per unit area when the circuit is shorted.
    • Related to how well the molecule absorbs light and generates charge carriers.
  13. Fill Factor (FF)

    • A measure of how “square” the current–voltage curve is.
    • Affects the efficiency of the solar cell.

Your Turn

Get the unique values and their counts for the columns construction, architecture, and complement.

Expected output:

--
construction
polymer     220
molecule    127
Name: count, dtype: Int64
--
architecture
bulk       270
DSSC        64
bilayer     13
Name: count, dtype: Int64
--
complement
PC61BM    139
PC71BM    133
TiO2       64
C60         9
ICBA        1
PDI         1
Name: count, dtype: Int64
# YOUR CODE HERE

Your Turn

Generate summary statistics (count, mean, standard deviation, min, quartiles, max) for all numerical columns in the dataset.

# YOUR CODE

Your Turn

Plot the data distribution for homo and lumo columns.

homo_lumo_dist
HOMO, LUMO distribution.
# YOUR CODE HERE

Your Turn

Generate a correlation matrix for the numerical columns in your dataset. Examine the strength and direction of these correlations, and reflect on what they might imply for your project.

# YOUR CODE HERE