refactor(!BREAKING): Group Manufacturers by Name, Fix Variant Typos (#19

)

* Refactor compile_to_csv.py script - change spelling varients to variants, move manufacturers folders into alphabetical group folders

* fix spelling for variant/variants in README.md, NB: resulting compiled CSV file - changed spelling for column is_varient to is_variant

* Move manufacturers folders into alphabetical group folders; it was done with group_folders.sh bash script which is attached in scripts folders

* update compile script to gracefully handle parsing errors, and fix some encoding issues

---------

Co-authored-by: Roman Lazunin <[email protected]>
Co-authored-by: Alex Toff <[email protected]>

README.md

@@ -1,6 +1,6 @@
 # About
 
-The aim of OpenAircraftType is to provide an open-source, adaptable database of aircraft types and varients.
+The aim of OpenAircraftType is to provide an open-source, adaptable database of aircraft types and variants.
 
 # Where is the data!
 
@@ -26,7 +26,7 @@ The folders are configurated as such:
         <ul>
         <li>**AIRCRAFT TYPE** Folder - This holds all of the information about the type. It contains a '**AIRCRAFT TYPE**.txt' file (more on this later)
           <ul>
-          <li>"Varients" Folder - This folder is optional, and contains any varients of the type.It contains a aircraft type information file for each varient</li>
+          <li>"Variants" Folder - This folder is optional, and contains any variants of the type.It contains a aircraft type information file for each variant</li>
           </ul>
         </li>
         </ul>
@@ -41,7 +41,7 @@ As an example:
         <li>A320
           <ul>
           <li>
-          	Varients
+          	Variants
             <ul><li>A320-111.txt</li><li>A320-121.txt</li><li>....txt</li></ul>
           </li>
           <li>A320.txt</li>
@@ -55,7 +55,7 @@ As an example:
 ### File Contents
 
 All of the files are in the format of a standard properties file, with the key being the first string
-(i.e `MODEL=A320`). If you want to add data that is not covered by the following properties, simply create a suitable, unique name for it and utilize it (making sure to add it to the appropriate section below). Ideally, all properties below are defined for each and every type (and/or varient)!
+(i.e `MODEL=A320`). If you want to add data that is not covered by the following properties, simply create a suitable, unique name for it and utilize it (making sure to add it to the appropriate section below). Ideally, all properties below are defined for each and every type (and/or variant)!
 
 ##### \*\*MANUFACTURER\*\*/manufacturer.txt
 This file contains details about the manufacturer. At the moment the following properties are in use in the database.
@@ -65,10 +65,10 @@ This file contains details about the manufacturer. At the moment the following p
 | NAME      	| The name of the manufacturer 			| Airbus 	|
 | COMPANY      	| The corporate name of the manufacturer    	| Airbus SE 	|
 
-##### \*\*MANUFACTURER\*\*/\*\*AIRCRAFT TYPE\*\*/\*\*AIRCRAFT TYPE\*\*.txt *or* \*\*MANUFACTURER\*\*/\*\*AIRCRAFT TYPE\*\*/Varients/\*\*VARIENT\*\*.txt
-Here, \*\*AIRCRAFT TYPE\*\* should ideally be the ICAO code of the aircraft type, and \*\*VARIENT\*\* should be the varient model name (e.g A320-111). However, it can in reality be whatever works. This file sets the base properties for the aircraft type (if it is the \*\*AIRCRAFT TYPE\*\*.txt file), or it overwrites/extends the aircraft type base file.
+##### \*\*MANUFACTURER\*\*/\*\*AIRCRAFT TYPE\*\*/\*\*AIRCRAFT TYPE\*\*.txt *or* \*\*MANUFACTURER\*\*/\*\*AIRCRAFT TYPE\*\*/Variants/\*\*VARIANT\*\*.txt
+Here, \*\*AIRCRAFT TYPE\*\* should ideally be the ICAO code of the aircraft type, and \*\*VARIANT\*\* should be the variant model name (e.g A320-111). However, it can in reality be whatever works. This file sets the base properties for the aircraft type (if it is the \*\*AIRCRAFT TYPE\*\*.txt file), or it overwrites/extends the aircraft type base file.
 
-As an example, when you define a varient, say AIRBUS/A320/Varients/A320-111.txt, any properties in this file will overwrite the properties of the AIRBUS/A320/A320.txt file for the varient.
+As an example, when you define a variant, say AIRBUS/A320/Variants/A320-111.txt, any properties in this file will overwrite the properties of the AIRBUS/A320/A320.txt file for the variant.
 
 | Property Key 	| Description   							| Example		|
 | --------------|:-----------------------------------------:|:-------------:|

compilers/scripts/compile_to_csv.py

@@ -1,60 +1,105 @@
 import os,csv,configparser
 
-outputcsv = "../build/ALL_DATA.csv";
-readdir = "../src";
-man_cols={"NAME":"manufacturer_name","COMPANY":"manufacturer_company"}
-type_cols={"MODEL":"model","ICAO":"icao","CLASS":"class","WAKE":"wake","ENG_TYPE":"engine_type","ENG_NUM":"engine_number","ENG_NAME":"engine_name","ENG_MAN":"engine_manufacturer","ENG_MODEL":"engine_model","ENG_THRUST":"engine_thrust","LENGTH":"length","WINGSPAN":"wingspan","TAIL_HEIGHT":"tail_height","RANGE":"range","CEILING":"ceiling","MAX_SPEED":"max_speed","PAX_CAP":"passenger_capacity","REMARKS":"remarks"}
-varient_cols={"BASE_MODEL":"base_model"}
-cols={**man_cols, **{"is_varient":'is_varient'}, **type_cols, **varient_cols}
-if os.path.exists(outputcsv):
-    os.remove(outputcsv)
-csv_file = open(outputcsv,'w', newline='')
-writer = csv.writer(csv_file);
-writer.writerow(cols.values())
-# Iterate Manufacturers
-for manu in os.listdir(readdir):
-    config = configparser.ConfigParser()
-    conf_file = open(readdir+"/"+manu+"/manufacturer.txt", 'r')
-    config.read_string('[MANUFACTURER]\n'+conf_file.read())
-    config = config['MANUFACTURER']
-    man_insert = [config.get('name'),config.get('company')]
-    print(config.get('name'))
-    # Iterate Types
-    for type in os.listdir(readdir+"/"+manu):
-        row = []
-        if os.path.isfile(os.path.join(readdir+"/"+manu, type)):
-            continue
-        model_config = configparser.ConfigParser()
-        model_conf_file = open(readdir+"/"+manu+"/"+type+"/"+type+".txt", 'r')
-        model_config.read_string('[MODEL]\n'+model_conf_file.read().replace('%','%%'))
-        model_config = model_config['MODEL']
-        model_insert =type_cols
-        for key in type_cols:
-            model_insert[key] = model_config.get(key, '')
-        writer.writerow(man_insert+[0]+list(model_insert.values()))
-        # Iterate Varients
-        if os.path.exists(readdir+"/"+manu+"/"+type+"/Varients"):
-            for varient in os.listdir(readdir+"/"+manu+"/"+type+"/Varients"):
-                if os.path.isdir(readdir+"/"+manu+"/"+type+"/Varients/"+varient):
-                    continue
-                varient_config = configparser.ConfigParser()
-                varient_config.optionxform = str
-                varient_conf_file = open(readdir+"/"+manu+"/"+type+"/Varients/"+varient, 'r')
-                varient_config.read_string('[VARIENT]\n'+varient_conf_file.read())
-                varient_config = varient_config['VARIENT']
-                #varient_insert = varient_cols
-                #for key in varient_cols:
-                #    varient_insert[key] = varient_config.get(key, '')
-                varient_insert2 = model_insert.copy()
-                for key in varient_config:
-                    if key in varient_insert2:
-                        varient_insert2[key] = varient_config[key]
-                writer.writerow(man_insert+[1]+list({**varient_insert2, **{'BASE_MODEL':model_config.get('MODEL', type)}}.values()))
-                varient_conf_file.close()
-
-        model_conf_file.close()
-    conf_file.close()
-    print('-----------')
+file_dir = os.path.dirname(__file__)
+output_csv = os.path.join(file_dir, "../../build/ALL_DATA.csv")
+read_dir = os.path.join(file_dir, "../../src")
+
+man_cols = {
+    "NAME": "manufacturer_name",
+    "COMPANY": "manufacturer_company"
+}
+
+type_cols = {
+    "MODEL": "model",
+    "ICAO": "icao",
+    "CLASS": "class",
+    "WAKE": "wake",
+    "ENG_TYPE": "engine_type",
+    "ENG_NUM": "engine_number",
+    "ENG_NAME": "engine_name",
+    "ENG_MAN": "engine_manufacturer",
+    "ENG_MODEL": "engine_model",
+    "ENG_THRUST": "engine_thrust",
+    "LENGTH": "length",
+    "WINGSPAN": "wingspan",
+    "TAIL_HEIGHT": "tail_height",
+    "RANGE": "range",
+    "CEILING": "ceiling",
+    "MAX_SPEED": "max_speed",
+    "PAX_CAP": "passenger_capacity",
+    "REMARKS": "remarks"
+}
+
+variant_cols = {"BASE_MODEL": "base_model"}
+
+cols = {**man_cols, **{"is_variant": 'is_variant'}, **type_cols, **variant_cols}
+
+if os.path.exists(output_csv):
+    os.remove(output_csv)
+
+csv_file = open(output_csv, 'w', newline='')
+
+writer = csv.writer(csv_file)
+writer.writerow(cols.values()) # write titles header row
+
+for group_dir in os.listdir(read_dir): # iterate high-level groups, like A, B, C etc.
 
-csv_file.close()
+    for manufacturer in os.listdir(os.path.join(read_dir, group_dir)): # Iterate Manufacturers
+        config = configparser.ConfigParser()
+
+        manufacturer_dir = os.path.join(read_dir, group_dir, manufacturer)
+
+        file_path = os.path.join(manufacturer_dir, "manufacturer.txt")
+        with open(file_path, 'r') as file:
+            try:
+                config.read_string('[MANUFACTURER]\n' + file.read())
+            except Exception as error:
+                print("Error whilst parsing " + os.path.abspath(file_path) + ": " + str(error))
+                exit()
+
+
+        config = config['MANUFACTURER']
+        man_insert = [config.get('name'), config.get('company')]
+        print(config.get('name'))
+
+        # Iterate Types
+        for aircraft_type in os.listdir(manufacturer_dir):
+
+            if os.path.isfile(os.path.join(manufacturer_dir, aircraft_type)):
+                continue
+            model_config = configparser.ConfigParser()
+
+            with open(os.path.join(manufacturer_dir, aircraft_type, aircraft_type + ".txt"), 'r') as file:
+                model_config.read_string('[MODEL]\n' + file.read().replace('%','%%'))
+
+            model_config = model_config['MODEL']
+            model_insert = type_cols
+
+            for key in type_cols:
+                model_insert[key] = model_config.get(key, '')
+
+            writer.writerow(man_insert + [0] + list(model_insert.values()))
+
+            # Iterate Variants
+            variants_dir = os.path.join(manufacturer_dir, aircraft_type, "Variants")
+            if os.path.exists(variants_dir):
+                for variant in os.listdir(variants_dir):
+                    if os.path.isdir(os.path.join(variants_dir, variant)):
+                        continue
+
+                    variant_config = configparser.ConfigParser()
+                    variant_config.optionxform = str
+
+                    with open(os.path.join(variants_dir, variant), 'r') as file:
+                        variant_config.read_string('[VARIANT]\n' + file.read())
+
+                    variant_config = variant_config['VARIANT']
+
+                    variant_insert = model_insert.copy()
+                    for key in variant_config:
+                        if key in variant_insert:
+                            variant_insert[key] = variant_config[key]
+                    writer.writerow(man_insert + [1] + list({**variant_insert, **{'BASE_MODEL': model_config.get('MODEL', aircraft_type)}}.values()))
+        print('-----------')
 
+csv_file.close()

compilers/scripts/group_folders.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+# Define the directory where your subfolders are located
+BASE_DIR="../src"
+
+# Iterate through each subfolder in the base directory
+for dir in "$BASE_DIR"/*/; do
+    # Get the first character of the directory name
+    folder_name=$(basename "$dir")
+    first_char="${folder_name:0:1}"
+
+    # Create the destination parent directory based on the first character
+    DEST_DIR="$BASE_DIR/$first_char"
+    mkdir -p "$DEST_DIR"
+
+    # Move the subfolder to the new parent directory
+    mv "$dir" "$DEST_DIR"
+done