Opened 6 years ago

Closed 6 years ago

Last modified 4 years ago

#814 closed design-issue (fixed)

how to validate data from HERON sources at staging time?

Reported by: mnair Owned by: mnair
Priority: major Milestone: heron-eldorado-update
Component: data-repository Keywords: public-web
Cc: rwaitman, achoudhary, badagarla, dconnolly Blocked By:
Blocking: Sensitive: no

Description

How about some scripts to do basic counts (patients, encounters, some vitals...)
when we stage data from sources such as O2/Epic?

see also investigation of flowsheet stats in ticket:789#comment:6

this is Dan, reporting for Mani

Change History (7)

comment:1 Changed 6 years ago by dconnolly

  • Owner changed from dconnolly to mnair
  • Status changed from new to assigned

Mani,

Would you like to elaborate a bit on this idea?

When you said "script," were you thinking of a series of steps for a person to do manually,
or an automated pass-fail test?

comment:2 Changed 6 years ago by achoudhary

  • Milestone changed from heron-clinton-update to heron-eldorado-update

I am doing is manual verification by looking into previous load logs.

comment:3 Changed 6 years ago by dconnolly

  • Cc dconnolly added

I'm trying to compare notes with John K. on the tumor registry data,
and it was a little tricky to figure out exactly which source data
went into the current build. It's in a feb directory where I was
expecting Jan.

The naaccr_feb_2012.log makes it clear enough, but maybe in the future we
should note the log file name and excerpt some basics such as start/end
date and number of rows in the ticket. e.g. ticket:836#comment:6

(For bonus points, I like to use md5sum to audit bulk data transfers;
e.g. 7b99781b8c847fde0af13b78a918a96d NAACCR_020112.DAT)

comment:4 Changed 6 years ago by rwaitman

We resolved that we will compare row numbers in the log files and put notes in the ticket for each release that verifies the counts are growing.

comment:5 Changed 6 years ago by rwaitman

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:6 Changed 4 years ago by dconnolly

  • Keywords public-web added

note source:heron_staging/Clarity_data_load/clarity_import_compare.py and hopes/plans for further automation in #2223.

comment:7 Changed 4 years ago by kcrane2

Cleared by Infosec

Note: See TracTickets for help on using tickets.