Skip to content

Commit 726d5b6

Browse files
eliSkylightericnagelhawley-skylight
authored
Err/app 473/update morbidity patient expected json (#767)
Co-authored-by: Eric Nagel <eric@ericnagel.com> Co-authored-by: Patrick Hawley <patrick.hawley@skylight.digital>
1 parent 2fc4c02 commit 726d5b6

6 files changed

Lines changed: 291 additions & 1196 deletions

File tree

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,9 @@ NEDSSDev/
6060
# Performance-testing
6161
venv/
6262
examples/
63+
64+
# node libs
65+
node_modules/
6366
.local/
6467
output/
6568

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,9 @@ To run only the `functional` tests in the reporting-pipeline-service
4747
./gradlew clean reporting-pipeline-service:test-functional
4848
```
4949

50+
### Adding MasterEtl-level Validation for Functional Tests
51+
Review [FuncationTestValidation.md](./documentation/FunctionalTestValidation.md) for more details.
52+
5053
### Windows Note (Testcontainers EOF Error)
5154

5255
If you see `unexpected EOF` while running containerized tests on Windows, this is usually caused by Testcontainers trying to copy the entire project directory into a container and hitting a locked file.

‎documentation/FunctionalTestValidation.md‎

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
# Functional Test Validation
2+
Currently, functional tests are validated using the changes captured on the RDB_MODERN database. This document provides instructions using the tools in this repository to capture the changes performed by MasterEtl on the legacy RDB database. These instructions assume your functional test files `query.sql` and `expected.json` enforce the functional test standards agreed upon by the team including the use of id fields (e.g., `local_id` or `act_uid`) and respective file syntax expected by `reporting-pipeline-service/src/test/java/gov/cdc/nbs/report/pipeline/integration/functional/DataDrivenFunctionalTests.java`. Additionally, `setup.sql` is expected to be completed, but may require changes based on your unique functional tests - <strong>those details are provided here</strong>.
3+
4+
This documentation was produced as a deliverable for the Jira Ticket [APP-473](https://cdc-nbs.atlassian.net/browse/APP-473).
5+
6+
## Process
7+
These instructions should be performed in the order displayed here.
8+
9+
### 1) Prepare test SQL inputs
10+
11+
1. Create temporary copies of `setup.sql`.
12+
2. Update all `last_chg_time` values to the current UTC timestamp using `GETDATE()`.
13+
3. (OPTIONAL) Increment all UIDs declared at the top of `setup.sql` by 1. This step will likely be needed if the instructions were not followed in order, requiring a <em>retry</em>. (ONLY FOR GENERATING DATA!).
14+
15+
### 2) Prepare for tracing and execute SQL in sequence
16+
4. Run `trace_db_logical_changes.py` pointing to RDB.
17+
```shell
18+
python utilities/local-db-tracing/trace_db_logical_changes.py --database RDB --user sa --password PizzaIsGood33\!
19+
```
20+
5. Using `sqlcmd`, execute the SQL files in the most logical order (e.g., first patient, then morbidity report).
21+
```shell
22+
sqlcmd -S localhost,3433 -U sa -P "PizzaIsGood33\!" -b -C -i <xxx-myFunctionalTestStep>/setup.sql
23+
```
24+
### 3) Run MasterEtl and capture changes
25+
7. When the tracing program prompts you to press **ENTER**, run Master ETL.
26+
8. After MasterEtl has completed, press **ENTER** in the tracing program to capture changes.
27+
9. Fill in the prompts at your discretion.
28+
10. Review results in `logical_changes.md` or `logical_changes.json` to focus on relevant changes.
29+
30+
### 4) Update validation artifacts
31+
32+
11. Compare inserts of expected RDB tables to your `query.sql` and `expected.json` files.
33+
34+
i. Ignore the timestamp differences as `GETDATE()` is only used for ensuring MasterEtl picks up the records.
35+
36+
ii. If you modified the values of any ID fields in the copied `setup.sql` you can ignore those as well when comparing your existing validation to the changes.
37+
38+
iii. Carefully review which tables are populated by MasterEtl and be sure to add entries to your validation files for any <em>missing</em> tables.
39+
40+
### 5) Functional Tests
41+
Execute the functional tests and determine what modifications are needed!
42+
43+
## Example Using Morbidity Report Functional Test Suite
44+
### Using setup.sql to Create Data on ODSE
45+
1. The existing `setup.sql` file was copied and all inserts for `last_chg_time` for all tables where updated to use `GETDATE()`. Consider the truncated example below. Note that there were files to copy/modify in this case for creating a patient and creating their morbidity report!
46+
```sql
47+
-- dbo.Entity_id (ORIGINAL)
48+
INSERT INTO [dbo].[entity_id]
49+
([entity_uid],
50+
[entity_id_seq],
51+
[add_time],
52+
[assigning_authority_cd],
53+
[last_chg_time],
54+
...)
55+
VALUES (@dbo_Entity_entity_uid,
56+
1,
57+
N'2026-04-10T20:26:11.673',
58+
N'GA',
59+
N'2026-04-10T20:26:11.673',
60+
...
61+
-- dbo.Entity_id (TEMP COPY)
62+
INSERT INTO [dbo].[entity_id]
63+
([entity_uid],
64+
[entity_id_seq],
65+
[add_time],
66+
[assigning_authority_cd],
67+
[last_chg_time],
68+
...
69+
VALUES (@dbo_Entity_entity_uid,
70+
1,
71+
N'2026-04-23T22:34:46.000',
72+
N'GA',
73+
GETDATE(),
74+
...)
75+
```
76+
3. Example of incrementing the UIDs and Local ID (ONLY FOR GENERATING DATA!). <strong>You want to avoid this because it will require having to consider the UIDS in your original validation file which can become tedious so that you do not override them with the incremented ones</strong>.
77+
```sql
78+
-- ORIGINAL
79+
DECLARE @dbo_Entity_entity_uid bigint = 20100001
80+
DECLARE @dbo_Postal_locator_postal_locator_uid bigint = 20100011
81+
DECLARE @dbo_Tele_locator_tele_locator_uid bigint = 20100012
82+
DECLARE @dbo_Act_act_uid bigint = 20100013
83+
DECLARE @dbo_Act_act_uid_2 bigint = 20100014
84+
DECLARE @dbo_Act_act_uid_3 bigint = 20100015
85+
DECLARE @dbo_Act_act_uid_4 bigint = 20100016
86+
DECLARE @dbo_Act_act_uid_5 bigint = 20100017
87+
DECLARE @dbo_Act_act_uid_6 bigint = 20100018
88+
...
89+
DECLARE @dbo_Act_act_uid_15 bigint = 20100027
90+
DECLARE @dbo_Person_local_id nvarchar(40) = N'PSN20100000GA01'
91+
-- TEMP COPY
92+
DECLARE @dbo_Entity_entity_uid bigint = 20100002
93+
DECLARE @dbo_Postal_locator_postal_locator_uid bigint = 20100012
94+
DECLARE @dbo_Tele_locator_tele_locator_uid bigint = 20100013
95+
DECLARE @dbo_Act_act_uid bigint = 20100028
96+
DECLARE @dbo_Act_act_uid_2 bigint = 20100029
97+
DECLARE @dbo_Act_act_uid_3 bigint = 20100030
98+
DECLARE @dbo_Act_act_uid_4 bigint = 20100031
99+
DECLARE @dbo_Act_act_uid_5 bigint = 20100032
100+
DECLARE @dbo_Act_act_uid_6 bigint = 20100033
101+
...
102+
DECLARE @dbo_Act_act_uid_15 bigint = 20100042
103+
DECLARE @dbo_Person_local_id nvarchar(40) = N'PSN20100000GA02'
104+
```
105+
4. `trace_db_logical_changes.py` was executed pointing to the RDB.
106+
```shell
107+
python utilities/local-db-tracing/trace_db_logical_changes.py --database RDB --user sa --password PizzaIsGood33\!
108+
```
109+
5. The following commands were executed to first create the patient, then create their morbidity report:
110+
```shell
111+
sqlcmd -S localhost,3433 -U sa -P "PizzaIsGood33\!" -b -C -i 010-addPatient/setup.sql
112+
```
113+
```shell
114+
sqlcmd -S localhost,3433 -U sa -P "PizzaIsGood33\!" -b -C -i 020-addMorbidityReport/setup.sql
115+
```
116+
6. Press **ENTER** in the python tracing program...
117+
7. Review the results in `utilities/local-db-tracing/output/20260423-183127-RDB/logical-changes.md`. Sample below:
118+
## 113. INSERT dbo.MORBIDITY_REPORT
119+
120+
| Metric | Value |
121+
| --- | --- |
122+
| Identity | business_keys: MORB_RPT_LOCAL_ID="OBS20100086GA01" |
123+
| Transaction end | 2026-04-23T22:22:34.720 |
124+
| LSN | 0x00006bf6000312400004 |
125+
126+
### Inserted Row
127+
128+
| Field | Value |
129+
| --- | --- |
130+
| DAYCARE_IND | "N" |
131+
| DIAGNOSIS_DT | "2026-04-05T00:00:00" |
132+
| DIE_FROM_ILLNESS_IND | "Y" |
133+
| ELECTRONIC_IND | "N" |
134+
| FOOD_HANDLER_IND | "N" |
135+
| HEALTHCARE_ORG_ASSOCIATE_IND | "UNK" |
136+
| HOSPITALIZED_IND | "Y" |
137+
| HSPTL_ADMISSION_DT | "2026-04-03T00:00:00" |
138+
| JURISDICTION_CD | "130001" |
139+
| JURISDICTION_NM | "Fulton County" |
140+
| MORB_RPT_CREATE_BY | 10009282 |
141+
| MORB_RPT_KEY | 3 |
142+
| MORB_RPT_LAST_UPDATE_BY | 10009282 |
143+
| MORB_RPT_LAST_UPDATE_DT | "2026-04-23T22:20:11.717" |
144+
| MORB_RPT_LOCAL_ID | "OBS20100086GA01" |
145+
| MORB_RPT_OID | 1300100009 |
146+
| MORB_RPT_OTHER_SPECIFY | "other something" |
147+
| MORB_RPT_SHARE_IND | "T" |
148+
| MORB_RPT_TYPE | "INIT" |
149+
| MORB_RPT_UID | 20100086 |
150+
| NURSING_HOME_ASSOCIATE_IND | "Y" |
151+
| PH_RECEIVE_DT | "2026-04-10T00:00:00" |
152+
| PREGNANT_IND | "Y" |
153+
| RDB_LAST_REFRESH_TIME | "2026-04-23T22:22:34.717" |
154+
| RECORD_STATUS_CD | "ACTIVE" |
155+
| SUSPECT_FOOD_WTRBORNE_ILLNESS | "N" |
156+
157+
### Updating `query.sql` and `expected.json`.
158+
The existing files for this functional test suite were accurate based on MasterEtl's output with the exception of 1 table: `MORBIDITY_REPORT`. This table was missing completely from validation! To account for this the following was performed:
159+
1. A new entry added to `query.sql` to get the columns modified on the table.
160+
```sql
161+
...
162+
-- 5: MORBIDITY_REPORT
163+
SELECT
164+
[DAYCARE_IND],
165+
[DIAGNOSIS_DT],
166+
[DIE_FROM_ILLNESS_IND],
167+
[ELECTRONIC_IND],
168+
[FOOD_HANDLER_IND],
169+
[HEALTHCARE_ORG_ASSOCIATE_IND],
170+
[HOSPITALIZED_IND],
171+
[HSPTL_ADMISSION_DT],
172+
[JURISDICTION_CD],
173+
[JURISDICTION_NM],
174+
[MORB_RPT_CREATE_BY],
175+
[MORB_RPT_KEY],
176+
[MORB_RPT_LAST_UPDATE_BY],
177+
[MORB_RPT_LAST_UPDATE_DT],
178+
[MORB_RPT_LOCAL_ID],
179+
[MORB_RPT_OID],
180+
[MORB_RPT_OTHER_SPECIFY],
181+
[MORB_RPT_SHARE_IND],
182+
[MORB_RPT_TYPE],
183+
[MORB_RPT_UID],
184+
[NURSING_HOME_ASSOCIATE_IND],
185+
[PH_RECEIVE_DT],
186+
[PREGNANT_IND],
187+
[RDB_LAST_REFRESH_TIME],
188+
[RECORD_STATUS_CD],
189+
[SUSPECT_FOOD_WTRBORNE_ILLNESS]
190+
FROM [RDB_MODERN].[dbo].[MORBIDITY_REPORT]
191+
WHERE [MORB_RPT_LOCAL_ID] = 'OBS20100027GA01';
192+
```
193+
2. A new entry added to `expected.json`.
194+
```json
195+
...
196+
"5": [
197+
{
198+
"DAYCARE_IND": "N",
199+
"DIAGNOSIS_DT": "2026-04-05T00:00:00.000",
200+
"DIE_FROM_ILLNESS_IND": "Y",
201+
"ELECTRONIC_IND": "N",
202+
"FOOD_HANDLER_IND": "N",
203+
"HEALTHCARE_ORG_ASSOCIATE_IND": "UNK",
204+
"HOSPITALIZED_IND": "Y",
205+
"HSPTL_ADMISSION_DT": "2026-04-03T00:00:00.000",
206+
"JURISDICTION_CD": "130001",
207+
"JURISDICTION_NM": "Fulton County",
208+
"MORB_RPT_CREATE_BY": 10009282,
209+
"MORB_RPT_LAST_UPDATE_BY": 10009282,
210+
"MORB_RPT_LAST_UPDATE_DT": "2026-04-10T20:26:11.853",
211+
"MORB_RPT_LOCAL_ID": "OBS20100027GA01",
212+
"MORB_RPT_OID": 1300100009,
213+
"MORB_RPT_OTHER_SPECIFY": "other something",
214+
"MORB_RPT_SHARE_IND": "T",
215+
"MORB_RPT_TYPE": "INIT",
216+
"MORB_RPT_UID": 20100027,
217+
"NURSING_HOME_ASSOCIATE_IND": "Y",
218+
"PH_RECEIVE_DT": "2026-04-10T00:00:00.000",
219+
"PREGNANT_IND": "Y",
220+
"RECORD_STATUS_CD": "ACTIVE",
221+
"SUSPECT_FOOD_WTRBORNE_ILLNESS": "N"
222+
}
223+
]
224+
```
225+
3. Execute the functional test suite!

0 commit comments

Comments
 (0)