Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#2298 closed defect (fixed)

Some REDCap fields are missing from the HERON concept tree

Reported by: badagarla Owned by: badagarla
Priority: major Milestone: heron-milford-update
Component: data-repository Keywords: public-web
Cc: ngraham, mhoag, tmcmahon, dconnolly Blocked By:
Blocking: 2377 Sensitive: no

Description

We discovered (ticket:2274#comment:5) that there are seven fields in ADC REDCap project that are missing labels.

We couldn't load the related concepts due to NULL name_char. Suggested solution is to make up labels for these fields using the field_name instead of element_label.

Change History (12)

comment:1 Changed 5 years ago by dconnolly

  • Component changed from REDCap to data-repository

comment:2 Changed 5 years ago by ngraham

  • Blocking set to 2329

comment:3 Changed 5 years ago by ngraham

comment:4 Changed 5 years ago by ngraham

  • Blocking 2329 deleted
  • Milestone changed from heron-solomon-update to heron-milford-update

It looks like the NULL field problem was mitigated in [148c8a526d68], [ae2662bb46b1], and [ef6780e5c007].

It looks like we added an exception to the "known concepts" test here [3abca266f3d9] (see also ticket:2295#comment:4).

So, I think it's safe to postpone this ticket until the next release.

comment:5 Changed 5 years ago by badagarla

  • Resolution set to fixed
  • Status changed from new to closed

[42c87e6af1c9] resolves the issue.

Tested this on current_medications

-- Check what the concepts and observations look like for current_medications
select * from blueherondata.observation_fact
where concept_cd like 'KUMCRC55|current_medications:%';
select * from blueherondata.concept_dimension
where concept_cd like 'KUMCRC55|current_medications:%';
select * from BLUEHERONMETADATA.redcap_terms
where c_basecode like 'KUMCRC55|current_medications%'
or c_fullname like '\i2b2\redcap\55\baseline_medical_history\current_medications\%';
/*
4	\i2b2\redcap\55\baseline_medical_history\current_medications\	Current Medications	N	FA 
5	\i2b2\redcap\55\baseline_medical_history\current_medications\10\	Blood glucose/sugar regulators	N	LA 
5	\i2b2\redcap\55\baseline_medical_history\current_medications\11\	Blood Thinners	N	LA 
*/

update redcap.redcap_metadata
set element_label = NULL
where element_label='Current Medications'
and field_name = 'current_medications'
and project_id=55
and form_name='baseline_medical_history';

-- RUN HERON LOAD
-- RUN LOAD_ALL_CONCEPTS

-- Check what the concepts and observations look like for current_medications
select * from blueherondata.observation_fact
where concept_cd like 'KUMCRC55|current_medications:%';
select * from blueherondata.concept_dimension
where concept_cd like 'KUMCRC55|current_medications:%';
select * from BLUEHERONMETADATA.redcap_terms
where c_basecode like 'KUMCRC55|current_medications%'
or c_fullname like '\i2b2\redcap\55\baseline_medical_history\current_medications\%';
/*
4	\i2b2\redcap\55\baseline_medical_history\current_medications\	current_medications
*/

For future purposes, took out the field_label for current_medications in the test redcap pid 55.

comment:6 Changed 5 years ago by dconnolly

  • Blocking set to 2377
  • Resolution fixed deleted
  • Status changed from closed to reopened

Concept loading failed (ticket:2377#comment:4). I suspect this isn't working as designed.

comment:7 Changed 5 years ago by badagarla

This is a different issue. There is only one concept missing the name_char:
KUMCRC2181|disch_disp:0

Debugging:

select * from redcap.redcap_metadata
where field_name='disch_disp'
order by to_number(option_id);

This shows that for some reason redcap_metadata.sql did not split this select field up properly.

changeset:c421284a1174

select project_id, field_name, option_id
from redcap.redcap_metadata_old
minus
select project_id, field_name, option_id
from redcap.redcap_metadata;
PROJECT_ID             FIELD_NAME                                                                                           OPTION_ID                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
---------------------- ---------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 
2181                   disch_disp                                                                                           0                                                                                          

comment:8 Changed 5 years ago by ngraham

  • Resolution set to fixed
  • Status changed from reopened to closed

comment:9 Changed 5 years ago by badagarla

Doing the same on [new id server] for the single sid ETL (w Nathan)

Verified that the issue exists on [new id server] also

select * from redcap.redcap_metadata
where field_name='disch_disp'
order by to_number(option_id);

Checked the ETL log to make sure it is loading something other than REDCap

badagarla@wXXX:/home/ngraham/bmidev/heron_load> python etl_log_review.py heron_load-2013-11-13.log | less
2013-11-13 11:25:12 [0/1157]   _event1157
2013-11-13 11:25:12 [1157/1162]     _event1157.Epic_encounter_diagnosis_observations
2013-11-13 11:25:12 [1162/1163]       _event1157.Epic_encounter_diagnosis_observations.id
2013-11-13 11:31:03 [1163/1223]         _event1157.Epic_encounter_diagnosis_observations.id.i2b2_facts_deid
2013-11-13 11:31:03 [1223/1232]           _event1157.Epic_encounter_diagnosis_observations.id.i2b2_facts_deid.Chunk
2013-11-13 11:31:36 [1232/1235]             line 87

Ran redcap_metadata.sql to rectify the issue.

comment:10 Changed 5 years ago by badagarla

Forgot to mention in the earlier comments: The test data on test server (1) REDCap to replicate this issue.

comment:11 Changed 5 years ago by ngraham

comment:12 Changed 5 years ago by mhoag

  • Keywords public-web added
  • Sensitive unset
Note: See TracTickets for help on using tickets.