old main

Staggering SIS Framework Copies

Situation: Concurrent copies of the same course template returns DEBUG and other messages in Q2 2016.

 
Sometimes we need to make a large number of copies of a template course, and therefore we need to stagger the data pushes. We would prefer to not have to manually push the copies individually.
Process:
 
  1. Generate list of courses to copy.  Note: in Macs, we might need to paste the generated copy.txt file into a new file using VI, or the line breaks might be in the wrong format.
  2. Split the copy list.
  3. Insert header line into the split files
  4. Launch the script that will push the individual feed files. This script sleeps for a few minutes between each data push.
 
 
Generate Feed File with all the courses to copy.  Spring 2017 Chemistry lab copies resulted in 83 lines in the file, called “copy.txt”.
 
In Linux/Mac, the split command will split a file into multiple files. The default is 1000 lines, using the -l attribute, we can change that. By default, the resulting files will be in the name format xaa, xab, xac, xad, etc…
This x can be changed with “split file.txt newfile”, where the resulting files would be newfileaa, newfileab, newfileac, etc…
 
I used:
 
split -l 1 copy.txt
 
and ended up with
 
xaa     xag     xam     xas     xay     xbe     xbk     xbq     xbw     xcc     xci     xco     xcu     xda
xab     xah     xan     xat     xaz     xbf     xbl     xbr     xbx     xcd     xcj     xcp     xcv     xdb
xac     xai     xao     xau     xba     xbg     xbm     xbs     xby     xce     xck     xcq     xcw     xdc
xad     xaj     xap     xav     xbb     xbh     xbn     xbt     xbz     xcf     xcl     xcr     xcx     xdd
xae     xak     xaq     xaw     xbc     xbi     xbo     xbu     xca     xcg     xcm     xcs     xcy     xde
xaf     xal     xar     xax     xbd     xbj     xbp     xbv     xcb     xch     xcn     xct     xcz
Add header to these individual line files
 
add_header.sh:
#!/bin/sh
# from http://stackoverflow.com/questions/13196993/adding-header-into-multiple-text-files
tmp=$(mktemp)   # Create a temporary file
trap "rm -f $tmp; exit 1" 0 1 2 3 13 15
header="EXTERNAL_COURSE_KEY,COURSE_ID,COURSE_NAME,TEMPLATE_COURSE_KEY,TERM_KEY,DATA_SOURCE_KEY"
for file in "$@"
do
   {
   echo "$header"
   cat $file
   } > $tmp
   mv $tmp $file
done
rm -f $tmp
trap 0
 
Using
 
./add_header.sh x*
 
Now I have 83 individual feed files, with the proper header.
 
Upload feed files
 
wait_run.sh:
 
#!/bin/bash
#------- Courses --------
#Processes Course Input File and pushes that to Bb Learn
CFILES=/Users/chris/Desktop/CHEMcopy/x*
for c in $CFILES
do
  echo -e "\nUploading $c file to Blackboard Learn..."
  # take action on each file. $c store current file name
test -f "$c" || continue
curl -k -H "Content-Type:text/plain" -u username:password --url https://bbserver/webapps/bb-data-integration-flatfile-BBLEARN/endpoint/course/store --data-bin @$c
mv $c done
sleep 240
done
This looks for all files beginning with “x” in /Users/chris/Desktop/CHEMcopy/.  Each file is uploaded, file is moved to a “done” folder underneath /Users/chris/Desktop/CHEMcopy, the scripts waits for 4 minutes, then the next file is uploaded. Rinse. Repeat.
]]>