Vocabulary Overhaul

In order to implement #253, we needed to turn our current dynamic
vocabularies system on its head.

Originally we had a single class that would impersonate all the
dynamically generated vocabularies.  Now we actually create a proper
class for each dynamically generated library and even create OMOP and
GDM-based variants.

Now that each vocabulary is properly represented as a class, we can
modify individual classes to behave as we'd like, so we should be able
to get dynamically generated operators to include lab values much more
easily now.

As with many large overhauls, I ran into other issues that I addressed
along the way:

- Validations we're working correctly and I've addressed an issue where
    identical warnings could stack up within the same operator
- ICD10 is now properly a multi-vocabulary operator
- Removed all hard-coded vocabulary-related operators except for Read
- Put existing behaviors under ConceptQL::Behaviors
- Reworked how requiring operators works

config/multiple_vocabularies.csv

@@ -1,3 +1,5 @@
 operator,vocabulary_id,domain
 CPT or HCPCS,CPT4,procedure_occurrence
 CPT or HCPCS,HCPCS,procedure_occurrence
+ICD10,ICD10CM,condition_occurrence
+ICD10,ICD10WHO,condition_occurrence

config/vocabularies.csv

@@ -6,30 +6,32 @@ BETOS,1000002,BETOS,,,,
 Clinical Code Type,1000004,Clinical Code Type,,,,
 Concept Class,68,OMOP Concept Class,,,Y,
 Condition Type,37,OMOP Condition Occurrence Type,,condition_occurrence,Y,
-CPT4,4,Current Procedural Terminology version 4 (AMA),CPT,procedure_occurrence,Y,"^\w{5}$"
+CPT4,4,Current Procedural Terminology version 4 (AMA),CPT,procedure_occurrence,,"^\w{5}$"
 CS_Schema,1000003,CS_Schema,,,,
 Currency,65,International Currency Symbol (ISO 4217),,,Y,
 Death Type,45,OMOP Death Type,,death,Y,
 Device Type,63,OMOP Device Type,,observation,Y,
 Domain,59,OMOP Domain,,,Y,
-DRG,40,Diagnosis-related group (CMS),,procedure_occurrence,Y,
+DRG,40,Diagnosis-related group (CMS),DRG,procedure_occurrence,,
 DrugTextString,26400,Drug Text String,Drug Text String,drug_exposure,,
 Drug Type,36,OMOP Drug Exposure Type,,drug_exposure,Y,
 DURATION_UNIT,1000005,DURATION_UNIT,,,,
 Ethnicity,44,OMOP Ethnicity,,person,Y,
+Gemscript,56,,Gemscript,drug_exposure,,"^\d{8}$"
 Gender,12,OMOP Gender,,person,Y,
 HCFASPEC,1000006,HCFASPEC,,,,
 HCFATYPE,1000007,HCFATYPE,,,,
-HCPCS,5,Healthcare Common Procedure Coding System (CMS),,procedure_occurrence,Y,"^\w{5}$"
+HCPCS,5,Healthcare Common Procedure Coding System (CMS),HCPCS,procedure_occurrence,,"^\w{5}$"
 ICD03_Morphology,1000008,ICD03_Morphology,,,,
 ICD03_Topography,1000009,ICD03_Topography,,,,
-ICD10,34,"International Classification of Diseases, Tenth Revision (WHO)",ICD-10 WHO,condition_occurrence,Y,"^\w\d\w($|.\w{1,4})$"
-ICD10CM,70,"International Classification of Diseases, Tenth Revision, Clinical Modification (NCHS)",ICD-10 CM,condition_occurrence,Y,"^\w\d\w($|.\w{1,4})$"
-ICD9CM,2,"International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 1 and 2 (NCHS)",ICD-9 CM,condition_occurrence,Y,"^(V\d{2}(\.\d{1,2})?|\d{3}(\.\d{1,2})?|E\d{3}(\.\d)?)$"
-ICD9Proc,3,"International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 3 (NCHS)",ICD-9 Proc,procedure_occurrence,Y,"^\d{2}(.\d{1,2})?$"
-LOINC,6,Logical Observation Identifiers Names and Codes (Regenstrief Institute),,observation,Y,"^\w{5}-\w$"
+ICD10,34,"International Classification of Diseases, Tenth Revision (WHO)",ICD-10 WHO,condition_occurrence,,"^\w\d\w($|.\w{1,4})$"
+ICD10PCS,35,"ICD-10 Procedure Coding System (CMS)",ICD-10 PCS,procedure_occurrence,,"^\w{7}$"
+ICD10CM,70,"International Classification of Diseases, Tenth Revision, Clinical Modification (NCHS)",ICD-10 CM,condition_occurrence,,"^\w\d\w($|.\w{1,4})$"
+ICD9CM,2,"International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 1 and 2 (NCHS)",ICD-9 CM,condition_occurrence,,"^(V\d{2}(\.\d{1,2})?|\d{3}(\.\d{1,2})?|E\d{3}(\.\d)?)$"
+ICD9Proc,3,"International Classification of Diseases, Ninth Revision, Clinical Modification, Volume 3 (NCHS)",ICD-9 Proc,procedure_occurrence,,"^\d{2}(.\d{1,2})?$"
+LOINC,6,Logical Observation Identifiers Names and Codes (Regenstrief Institute),,observation,,"^\w{5}-\w$"
 Meas Type,64,OMOP Measurement Type,,observation,Y,
-NDC,9,National Drug Code (FDA and manufacturers),,drug_exposure,Y,
+NDC,9,National Drug Code (FDA and manufacturers),,drug_exposure,,
 Note Type,58,OMOP Note Type,,,Y,
 NUCC,47,National Uniform Claim Committee Health Care Provider Taxonomy Code Set (NUCC),,,Y,
 Observation Type,39,OMOP Observation Type,,observation,Y,
@@ -43,7 +45,8 @@ Procedure Type,38,OMOP Procedure Occurrence Type,,procedure_occurrence,Y,
 PROVIDER_FACILITY_TYPE,1000014,PROVIDER_FACILITY_TYPE,,,,
 Race,13,Race and Ethnicity Code Set (USBC),,person,Y,
 Relationship,66,OMOP Relationship,,,Y,
-RxNorm,8,RxNorm (NLM),,drug_exposure,Y,
+Revenue Code,43,UB04/CMS1450 Revenue Codes (CMS),Revenue Code,procedure_occurrence,,
+RxNorm,8,RxNorm (NLM),,drug_exposure,,
 SEER_ADJAJC6M,1000015,SEER_ADJAJC6M,,,,
 SEER_ADJAJC6N,1000016,SEER_ADJAJC6N,,,,
 SEER_ADJAJC6T,1000017,SEER_ADJAJC6T,,,,
@@ -68,7 +71,7 @@ SEER_SXPRIF,1000035,SEER_SXPRIF,,,,
 SEER_SXSITF,1000036,SEER_SXSITF,,,,
 SEER_TYPEFU,1000037,SEER_TYPEFU,,,,
 SEER_URBRUR,1000038,SEER_URBRUR,,,,
-SNOMED,1,SNOMED-CT,SNOMED-CT,condition_occurrence,Y,
+SNOMED,1,SNOMED-CT,SNOMED-CT,condition_occurrence,,
 Specialty,48,Medicare provider/supplier specialty codes (CMS),,provider,Y,
 STUS_CD,1000039,STUS_CD,,,,
 TYPESRVC,1000040,TYPESRVC,,,,

lib/conceptql.rb

@@ -2,10 +2,13 @@
 require "conceptql/logger"
 require "conceptql/paths"
 require "conceptql/utils"
+require "conceptql/operators/operator"
+require "conceptql/behaviors/code_lister"
 require "conceptql/behaviors/timeless"
 require "conceptql/behaviors/unwindowable"
 require "conceptql/behaviors/windowable"
 require "conceptql/behaviors/utilizable"
+require "conceptql/vocabularies/dynamic_vocabularies"
 require "conceptql/query"
 require "conceptql/null_query"
 require "conceptql/database"
@@ -42,15 +45,26 @@ def self.metadata(opts = {})
 
   def self.categories
     [
-      'Select by Clinical Codes',
-      'Select by Property',
-      'Get Related Data',
-      'Modify Data',
-      'Combine Streams',
-      'Filter by Comparing',
-      'Filter Single Stream',
+      "Select by Clinical Codes",
+      "Select by Property",
+      "Get Related Data",
+      "Modify Data",
+      "Combine Streams",
+      "Filter by Comparing",
+      "Filter Single Stream",
     ].map.with_index do |name, priority|
       { name: name, priority: priority }
     end
   end
 end
+
+# Require all operator subclasses eagerly
+#
+# First, require vocabulary operator.  It will establish operators for all
+# vocabularies found in Lexicon.  Then other operators might override
+# some of those dynamically generated operators
+ConceptQL::Vocabularies::DynamicVocabularies.new.register_operators
+Dir.new(File.dirname(__FILE__) + "/conceptql/operators").
+  entries.
+  each{|filename| require_relative "conceptql/operators/" + filename if filename =~ /\.rb\z/ && filename != File.basename(__FILE__)}
+ConceptQL::Operators.operators.values.each(&:freeze)

lib/conceptql/behaviors/code_lister.rb

@@ -0,0 +1,21 @@
+module ConceptQL
+  module Behaviors
+    module CodeLister
+      ConceptCode = Struct.new(:vocabulary, :code, :description) do
+        def to_s
+          if description
+            "#{vocabulary} #{code}: #{description}"
+          else
+            "#{vocabulary} #{code}"
+          end
+        end
+      end
+
+      def code_list(db)
+        describe_codes(db, arguments).map do |code, desc|
+          ConceptCode.new(preferred_name, code, desc)
+        end
+      end
+    end
+  end
+end

lib/conceptql/behaviors/drugish.rb

@@ -1,11 +1,13 @@
 module ConceptQL
-  module Drugish
-    def self.included(base)
-      base.require_column(:drug_name)
-      base.require_column(:drug_amount)
-      base.require_column(:drug_amount_units)
-      base.require_column(:drug_quantity)
-      base.require_column(:drug_days_supply)
+  module Behaviors
+    module Drugish
+      def self.included(base)
+        base.require_column(:drug_name)
+        base.require_column(:drug_amount)
+        base.require_column(:drug_amount_units)
+        base.require_column(:drug_quantity)
+        base.require_column(:drug_days_supply)
+      end
     end
   end
 end

lib/conceptql/behaviors/labish.rb

@@ -1,13 +1,14 @@
 module ConceptQL
-  module Labish
-    def self.included(base)
-      base.require_column :value_as_number
-      base.require_column :value_as_string
-      base.require_column :value_as_concept_id
-      base.require_column :unit_source_value
-      base.require_column :range_low
-      base.require_column :range_high
+  module Behaviors
+    module Labish
+      def self.included(base)
+        base.require_column :value_as_number
+        base.require_column :value_as_string
+        base.require_column :value_as_concept_id
+        base.require_column :unit_source_value
+        base.require_column :range_low
+        base.require_column :range_high
+      end
     end
   end
 end
-

lib/conceptql/cli.rb

@@ -49,6 +49,7 @@ def run_statement(statement_file)
     def sql(statement_file)
       q = cdb(options).query(criteria_from_file(statement_file))
       puts q.sql(:formatted, :create_tables)
+      puts q.query.from_self.group_and_count(:criterion_domain).order(:criterion_domain).sql
     end
 
     desc 'code_list statement_file', 'Reads the ConceptQL statement from the statement file and prints the code list out'